I'm trying to combine data from multiple excel columns into one column and want to avoid copying and pasting it manually over and over each time I receive a file.
The data will come in this format:
C1 C2 C3 Date1 Date2 Date3 ... DateN
text text2 text3 ### ### ### ###
text text2 text3 ### ### ### ###
The end goal is to have all the dates in a column by themselves and to have all the values in a column next to their respective dates to load into a MSS table like so:
C1 C2 C3 Results Date
text text2 text3 ### Date1
text text2 text3 ### Date1
text text2 text3 ### Date2
text text2 text3 ### Date2
text text2 text3 ### Date3
text text2 text3 ### Date3
So far I'm drawing a blank on what could be done to automate this.
You can use SSIS Unpivot Transformation; it does exactly what you need. The only caveat - number of Date columns has to be fixed before you design and run SSIS package; you cannot handle dynamic number of columns in SSIS with builtin transformations.
Related
I have cells with 4 pieces of information in each cell.
Each piece of information has a label.
I'm trying to extract the 4 pieces of information (without the labels) into the next 4 cells.
Results I'm Looking for:
Columns E-H to list the corresponding data (pulled from Column D), without the Labels (Calendar:, Start Time:, End Time:, Details:)
EXAMPLE:
COLUMN D
|Calendar: 02/08/2019, Start Time: 12:00 PM, End Time: 10:00 PM, Details: Birthday|
Should break into COLUMN E - H as
| 02/08/2019 | 12:00 PM | 10:00 PM | Birthday|
I found this solution, it is ugly and long - but it works!
=SPLIT(SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(D9,"Calendar: ","^"),", Start Time: ","^"),", End Time: ","^"),", Details: ","^"),"^",TRUE,TRUE)
Im kinda new to all the SSIS stuff. And im stuck with it. i want to combine multiple CSV files and then put them into a database. The files all have the same info. Examples:
File 1
Week Text1
22-10-2018 58
29-10-2018 12
File 2
Week Text2
22-10-2018 55
29-10-2018 48
File 3
Week Text3
22-10-2018 14
29-10-2018 99
Expected result:
Result in DB
Week Text1 Text2 Text3
22-10-2018 58 55 14
29-10-2018 12 48 99
I got this far by selecting the documents, use a sort and then a join merge. For 3 documents this took me 3 sorts and 2 join merge's. I have to do this for about 86 documents.. there has to be an easier way.
Thanks in advance.
I agree with KeithL, I recommend that your final table look like this:
Week Outcome Value DateModified
=======================================================
22-10-2018 AI 58 2018-10-23 20:49
29-10-2018 AI 32 2018-10-23 20:49
22-10-2018 Agile 51 2018-10-23 20:49
29-10-2018 Agile 22 2018-10-23 20:49
If you want to pivot Weeks or outcomes, do it in your reporting tool.
Don't create tables with dynamic named columns - that's a bad idea
Anyway here is an approach that uses a staging table.
Create a staging table that your file will be inserted into:
Script 1:
CREATE TABLE Staging (
[Week] VARCHAR(50),
Value VARCHAR(50),
DateModified DATETIME2(0) DEFAULT(GETDATE())
)
Import the entire file in, including headings. In other words, when defining the file format, don't tick 'columns in first row'
We do this for two reasons:
SSIS can't import files with with different heading names using the same data flow
We need to capture the heading name in our staging table
After you import a file your staging table looks like this:
Week Value DateModified
=======================================
Week Agile 2018-10-23 20:49
22-10-2018 58 2018-10-23 20:49
29-10-2018 32 2018-10-23 20:49
Now select out the data in the shape we want to load it in. Run this in your database after importing the data to check:
Script 2:
SELECT Week, Value,
(SELECT TOP 1 Value FROM Staging WHERE Week = 'Week') Outcome
FROM staging
WHERE Week <> 'Week'
Now add an INSERT and some logic to stop duplicates. Put this into an execute SQL task after the data import
Script 3:
WITH SRC As (
SELECT Week, Value,
(SELECT TOP 1 Value FROM Staging WHERE Week = 'Week') Outcome
FROM staging As SRC
WHERE Week <> 'Week'
)
INSERT INTO FinalTable (Week,Value, Outcome)
select Week, Value, Outcome
FROM SRC
WHERE NOT EXISTS (
SELECT * FROM FinalTable TGT
WHERE TGT.Week = SRC.Week
AND TGT.Outcome = SRC.Outcome
)
Now you wrap this up in a for each file loop that repeats this for each file in the folder. Don't forget that you need to TRUNCATE TABLE staging before importing each file.
In Summary:
Set up a for each file iterator
Inside this goes:
A SQL Task with TRUNCATE TABLE Staging;
A data flow to import the text file from the iterator into the staging table
A SQL Task with Script 3 in it
I've put the DateModified columns in the tables to help you troubleshoot.
Good things: you can run this over and over and reimport the same file and you won't get duplicates
Bad thing: Possibility of cast failures when inserting VARCHAR into DATE or INT
You can read your file(s) using a simple C# script component (Source).
You need to add your 3 columns to output0.
Week as DT_Date
Type as DT_STR
Value as DT_I4
string[] lines = System.IO.File.ReadAllLines([filename]);
int ctr = 0;
string type;
foreach(string line in lines)
{
string[] col = line.Split(',');
if(ctr==0) //First line is header
{
type = col[1];
}
else
{
Output0Buffer.AddRow();
Output0Buffer.Week = DateTime.Parse(col[0]);
Output0Buffer.Type = type;
Output0Buffer.Value = int.Parse(col[1]);
}
ctr++;
}
After you load to a table you can always create a view with a dynamic pivot.
I'm having issues with multi-value parameters. I was expecting I could load distinct values for the label and value combination from the default dataset but it's not working.
I have a query that looks like this:
Id Name Col1 Col2
1 Foo aaa ddd
1 Foo xxx sss
1 Foo yyy zzz
2 Bar bbb eee
3 Huh ccc fff
And want to grab the ID - Name combo from the dataset. Ideally, the results would be:
Value Label
1 Foo
2 Bar
3 Huh
However, the actual results are:
Value Label
1 Foo
1 Foo
1 Foo
2 Bar
3 Huh
EDIT: I consider this a poor solution, but made a new dataset only to collect the fields as I needed them and populate the parameter dropdown list. It's not the prettiest outcome, but should work until I find a proper way to do it.
Some of my colleagues are telling me this is the only way to do it, so I have to create a new data set every time I want to fill a new multi-value parameter from a query... Please tell me if it's actually the way to do it, I find it extremely inefficient.
You have to create a dataset for each parameter in order to feed it properly.
I am new to SSRS and need some help.
I am using a Matrix to Pivot my data.
Dataset looks like this:
TEST TOOLID
----- ------
TEST1 123
TEST1 234
TEST2 456
What I currently have is this:
TEST1 TEST2
TOOL ID 123 N/A
TOOL ID 234 N/A
TOOL ID N/A 456
The output I am looking for is (NULLS should be converted to N/A):
TEST1 TEST2
TOOL ID 1 123 456
TOOL ID 2 234 N/A
Thanks
jlimited
Choose Details in lower left of view pane under 'Row Groups'
Right Click>Add Group>Parent Group
Choose your item you want to group by, include header is the standard behavior(check box to select). Hit okay.
You are now grouping your columns and you should see a bit more geometric ( in the gray space before the rows showing they are grouped accordingly now.
I have excel source and sql server table .
Excel Source column is
Mno Price1 Price2
111 10 20
222 30 25
333 40 30
444 34 09
555 23 abc
Sql server Table
Product table name
PId Mno Sprice BPrice
1 111 3 50
2 222 14 23
3 444 32 34
4 555 43 45
5 666 21 67
I want to compare excel source Mno(Model number) with sql server Product table Mno (Model number), and if it is same i want to update Sql server Product table SPrice and Bprice.
Please tell me what are the steps i want to do?
I want to validate that excel sheet also, because in excel Price2 column have string values
if it's string value i want to send mail which row data are wrong.
I am new for SSIS so please give me details.
Read your new data in a source, use a lookup component for existing data. Direct row matches to a oledb command for update, and a destination for your non-matches for inserts (if you want to enter new products).
Personally I think the simplest way to do this is to use a dataflow to bring the excel file into a staging table and do any clean up if need be. Then as the next step inmteh control flow have an Execute SQl task that does the update. Or if you need either an update or an insert if the record is new, use a Merge statement in the Execute SQl task.
You can use a Merge Join Transformation with a full outer join (remember to sort your datasets before they input to the Merge Join Transformation), then have the output go to a Conditional Split Transformation. The Conditional Split Transformation can determine whether or not a row needs to be updated, inserted, or deleted and direct the flow to the appropriate transform to do that.
This was off the top of my head, and there may be a simpler transform for this. I haven't had the opportunity to work with SSIS in almost a year, so I might be getting a bit rusty.