I have an excel file that I imported in SSIS and want to transfer it to SQL server as a new table in the destination DB, the table is shown in the screenshot below:
I want to calculate the Average of Scores for each player as new column as shown in the screenshot.
I tried to accomplish this, but I couldn't success.
Rather let the database handle the transformation as your database will most likely do it faster than letting SSIS do it, insert your data into your destination but leave average column as NULL, write an UPDATE sql or STORED PROCEDURE to update your average column after the load, let SSIS call the update after doing the load
Related
I have various data inflows from different servers. I need to populate all the data in one single destination table.
I need a staging table inside the data flow task where i can insert all the data and do some validations and then insert those data directly to destination table.
Is this possible ? Also i need to pull only the recent records based on the modified date column of the source table. (I'm trying to create a variable that stores the date of the recent successful package execution which I can compare it with Modified date column) Is this possible ?
Let me know is there any best way to achieve above all process.
I don't know that it's possible to do this within one data flow task, but couldn't you use one data flow task for each stage and link them via control flow instead?
It sounds to me like the best way to do what you want is to import the data and then run a SQL task on the table with the imported data (control flow object in SSIS), it should be easy to filter the source table as you mentioned:
SELECT *
INTO newTable
FROM sourceTable
WHERE DateModified >= somedate
Hope that is useful.
I have a question which I'm sure has been asked before, but I don't know the terminology for this question. (Hence, I have tried searching for an answer to this question on this, but no luck.)
I am on SQL Server Management Studio 2008. I have a table that was created by an import of a flat file. At the beginning of every month, I want to update this table with a new version of the given flat file. The headers on the flat file / table will always stay the same, and no previous records will be lost. Data on previous records may change, and new records will be included.
What is the best way for doing this each month? Right now, my solution is to delete the current table and re-create it with an import of the new flat file. (Or, I could run a truncate, and then re-import.)
One of the faster methods would be to drop all indexes, truncate, re-import, and re-create all indexes. Note that with a flat file you could automate using SSIS, or you could use a BULK INSERT for a job schedule. For instance, if the file is in the same location every month and all the delimiters and details are the same, a procedure or TSQL script that BULK INSERTs the file would work when called by a job once a month on a schedule.
BULK INSERT MonthlyTable
FROM 'C:\MonthlyFileDrop\MonthlyFile.txt'
WITH (
FIELDTERMINATOR = ','
,ROWTERMINATOR = '0x0a'
,FIRSTROW=2
)
Another approach (one that I'm not partial to) would be to insert the data into a stage table, compare what data are not in the existing table from the staging table, populate those data, then re-index the existing table and drop the staging table.
I am importing an Excel table from an ftp site to SQL using SSIS. The destination table is going to be used to calculate good and bad records based on another SQL database. Here is my problem. The Excel file is name RTW_032613_ABC_123.xls. This file name is a concatenation of a number of fields. I cannot recreate it based on the fields in the table, so I need to retain it and pass it to the new table in SQL. I have a parameter #FileName that I am using to loop through the files in the ftp folder. What I would like to do is either combine the import of data from the Excel file with the file name or insert the file name in each record after the import. I am calling the SSIS procedure from another stored procedure in SQL. I tried adding a SQL data flow task but I am not seeing where I add the insert statement on either the SQL Server Compact Destination or SQL Server Destination.
I am over my head with SSIS on this one. The key is that the parameter that I need is available in SSIS but I really need to get it passed on to my SQL table.
TIA
If I'm reading your question right, you have an SSIS package with a variable containing the filename and you want to save the filename with each row that you are sending to your SQL table? If so:
Add a derived column to the data flow, making a new column and referencing the variable in the expression
Include that new column in the mapping for the destination of your data flow, sending the filename to whichever column you would like to save that data in.
No need for a seperate SQL task.
I have one excel sheet which has information which has to be saved in 4 tables. I have to create unique ids for each table.I also have to insert some data into table 1 and then the unique id created there will be used for inserting data into second table(Referential Integrity) . Moreover one table will always get records to be inserted but for rest 3 tables if some data already exists then it has to be updated and not inserted. I m new to SSIS so please guide me on how to proceed further in SSIS.
loads of requirements :)
First, here is an example of a package that loads an excel sheet to a sql database.
You can easily follow it to build your package.
Differences:
You say you need to insert the same data on 4 tables, so between your excel source and your destination, you will add a multicast component and them instead of 1 destination, you will have 4. The "multicast" will create 4 copies of your data, so you can insert into your 4 tables.
The IDs may be a problem, since the 4 destinations will execute separately, you cant get the ID inserted on the first table to update the second. I suggest you do it using a T-SQL on a "Execute SQL task" after everything is imported .
If that is not possible you will need to have 4 separately data flows where on each one you do the inserts reading from your excel and joining with the result of the previous insert with a lookup task
Import it into a Temp table on SQL server. Then you will be able to write a query which retrieves from the Temp table to multiple table.
Hope this solves your problem as per your requirement.
Rather new to SSIS so not sure how to handle this.
I have a flat file which i managed to successfully read from. So right now my data flow consists of just a flat file source.
What i want to do is something like this:
Update SqlTable S
set s.columnA = f.columna
from FlatFile f
where s.columnID = f.columnID
Right now the only way i can see of doing this would be to insert the contents of the flat file into a sql table, then doing my update. This seems wasteful considering i don't need to save the data of the flat file. I just need to update an existing sql table based on the data in the flat file. So is there some way to run the query directly in the SSIS package instead of having to insert a bunch of data into a sql table that i will just wind up dropping?
thanks
Update SqlTable S set s.columnA = f.columna from FlatFile f where s.columnID = f.columnID
That statement above is a SQL statement. You cannot connect a sql table to a flat file. You need to work in SQL to do an update, since that is where the table lives
You have 2 choices:
Use an OLEDB Command component within the data flow. The downside is this calls the statement for each record, so if you have 1,000s of records it is very inefficient.
Push the records to a table using an OLE DB Destination and then you can call your update using an Execute SQL Task. You can then truncate the table if you like
A possible 3rd option is to roll your own OLE DB destination to do an update on record sets vs records.
While this might sound wasteful, to create a table in the database to store update records, it is done very often. You just drop the worktable or truncate when complete.
You could add an OLE DB Command component to the Data Flow that retrieves data from the flat file. The OLE DB Command would do a single row update for each record retrieved from the flat file. This might be okay if there are few rows in the flat file; but, you can imagine how bad performance will be if there are many rows in the flat file.
I think you'll find that sending the flat file rows to a database table and running a single UPDATE is going to be the best performer for lots of data.
I haven't tried this but have you tried sending to a recordset destination and then running the update using that?
The bulk load into a temporary table is the way to go and then do your updates from the temp table. As a previous poster says it is quite a common aproach to stuff data into a staging area prior to doing some more work with the data and then dropping or truncating the table