I have a package where I have an input file that has a header line
TI,2
and detail line(s) that look like this
YP,302,,0000000000000061.00,20170714,CHK #9999,R04,9999
I have to do some processing on the detail lines. The file name is in a variable called User::FileName
In my Data Flow I have a conditional split where I shoot the Header Record to a path where I create a file with just the header record (it doesn't change).
I process all of the detail records. I have to go into SQL to do this and write out the results into a comma delimited flat file with the same name as the input file (using the variable).
So now I have a Header file with a fixed name and a detail file with the name in a variable. I need to combine these. I am trying to create a .BAT that that says
copy /y /d /b header.txt + User::FileName User::Filename (with the proper values substituted for the variable) and then execute this with an Execute Process task.
I am doing this with a Data Flow Task. The source is a Flat file (copy.bat) that contains 2 columns. Column 0 has copy /y /d /b. I have a derived column called Rest_of_Copy that has header.txt + User::FileName + " " + User::Filename
On the output file Destination I also have 2 columns. I am mapping Column 0 (the copy /y /d /b) to column 0 on the output file and The Derived Column Rest_of_Copy (which should contain the results of header.txt + User::FileName + " " + User::Filename). The Connection Manager for the Destination File is Copyout.bat
When I run the package Copyout.bat is empty.
I can't figure out why. Can anybody help?
I found a different way to do it. I wrote out the 2 files to fixed names, so the COPY command was fixed, and then renamed the output file to the SSIS variable using a File System task.
Thanks,
Dick
Related
I am producing two files from an SSIS package.
One is the main content and the other is the header.
After I have output both files - I am then merging them using an Execute Process Task.
So I have a content.txt and a header.txt.
/C copy /B \filepath\header.txt + \filepath\content.txt \filepath\result.txt
What I want to do at this stage it append the data to the result.txt so it becomes result_09102019.txt.
How do I achieve that within the snippet of code I have above?
I am not using an Execute Process Task now in order to achieve the filename.
Instead using an Execute SQL Task to simply write the result set to a variable - which I then point at me Flat File Connection.
select 'filename_' + format(getdate(), 'yyyyMMddHHmm') + '.csv'
The single row result set gets written to a variable called OutputFileName.
I then have an OutputFolder variable and then combine the OutputFolder and OutputFileName to another variable called OutputPath.
Output path is then added via an Expression to the file connection.
suggestion - explore the use of variables in building up file names.
Store filename in variable and create tables with the filename in SSIS
I received an SSIS package that was built by someone else. In one of the control folow that gets the data from oledb data source and puts it into a csv file. Their is a derived column between the oledb data source task and the flat file destination task.
The derived data flow task creates some new columns with some of the expression having an equal to sign.
These columns are then redirected to CSV file. This file is again read by another data flow task that reads this created CSV file. If I remove the = sign from the derived column it gives me an error while reading the flat file.
[Flat File Source 2] Error: The column delimiter for column "CompanyCode" was not found.
The data type of column company code is varchar.
My Question:
What is the = sign in the derived column task used for?
The equals sign is a string literal in the expression.
If your data had a row where the value of CompanyCode was
ABC
Then the derived column CompanyCodeWithZero would contain
="ABC"
I can't surmise why removing the equal sign results in the error you are getting since it seems to be coming from some other part of the package not described in your question (Flat File Source 2).
The error message you provided
[Flat File Source 2] Error: The column delimiter for column "CompanyCode" was not found.
Is showing that it has no relation with the derived column transformation, it is thrown by [Flat File Source 2] and it occur when the flat file used doesn't matches the settings used in the Flat File Connection manager, check that this file does not contains rows with a corrupted structure.
And about the equal sign (=) (like #TabAlleman said)
The equals sign is a string literal in the expression.
i wrote sql server table data to .csv file by using SSIS package. In flat file conncetion i've mentioned semi-colon as delimiter. Some of the values in my table are having commas(for example : CODE A,B,C) in .csv file CODE A is coming into 1st column and B is in 2nd vice-versa. now how can i keep all these values in a first column of .csv file with delimiter as ;
Probably you are opening your flat file in Excel, and opening it as a comma-separated file.
You need to specify to Excel when you open it that your file is delimited by semi-colons. Then it will display correctly.
I have a folder containing a number of csv files, e.g. "leeds dz.csv", "leeds gh.csv", "leeds fr.csv". The first part of the file names is constant (i.e. always "leeds").
I want to import each to Stata individually, convert to .dta file and save it. Currently I have this code:
cd "etcetc"
clear
local myfilelist : dir . files"*.csv"
foreach file of local myfilelist {
drop _all
insheet using `file', comma
local outfile = subinstr("`file'",".csv","",.)
save "`outfile'", replace
}
The code works fine if I rename all the .csv files manually to delete the "leeds" part, ie if each .csv is named "dz.csv" instead of "leeds dz.csv" etc.
However, if I do not do this deletion I receive the error "invalid 'dz.csv' "
I'm guessing this has something to do with my 3rd line of code, in particular the "*.csv". But I'm unsure how to adapt the code/ why it won't allow me to import files with a space in the name?
The line
insheet using `file', comma
will be problematic with any filename containing spaces.
Try
insheet using "`file'", comma
The help for insheet is quite explicit on this:
If filename is specified without an extension, .raw is assumed. If your
filename contains embedded spaces, remember to enclose it in double
quotes.
in my foreach loop container, I would like to delete the current processed file.
I try as follows, but no file is deleted at the end, any idea??
Here is the property of my loop , Current processed file comes from FileNameSimu variable
I would like to delete the current file
Make sure that the value in the variable User::FileNameSimu contains the file path like C:\Folder1\SubFolder2\File.txt and not just the file name File.txt
Please note the description of the property SourceVariable on File System Task. It expects a path.
On the Variables window, select the variable FilePath and press F4 to view the properties of the variable. Change the property EvaluateAsExpression to True and set the value #[User::Directory] + #[User::FileName] to the property Expression assuming that your variable Directory contains the folder path and the variable FileName contains the name. Make sure that the variable Directory ends with a backslash at the end like C:\temp\ and not like C:\temp. If it doesn't have a backslash at the end change your expression to #[User::Directory] + "\\" + #[User::FileName]
Or use the backslash in the expression