I have the following Sequence Container inside of a ForEach loop in my SSIS package:
I am busy testing the ROLLBACK TRANSACTION statement, it executes fine but it does not rollback.
Not sure if I am missing anything?
Thanks in advance.
EDIT:
This is how the data flow looks like in my Sequence Container:
I took an alternative route to make sure that my fail over works. What I did was I added a column to my tables that will store a GUID and when there is an error I just delete records from tables where that GUID is equal to the one in that is used in that session. I added the GUID in my select statement that my ForEach Container will use and then I write it to the table.
SELECT
ReconMedicalAidFile.fReconMedicalAidFileID
,ReconMedicalAidFile.fReconMedicalAidID
,ReconMedicalAids.fMedicalAidID
,ReconMedicalAidFile.fFileName
,ReconMedicalAidFile.fFileLocation
,ReconMedicalAidFile.fFileImportedDate
,ReconMedicalAidFile.fNumberRecords
,ReconMedicalAidFile.fUser
,ReconMedicalAidFile.fIsImported
,CONVERT(varchar(50),NEWID()) AS 'Session'
FROM ReconMedicalAidFile
INNER JOIN ReconMedicalAids ON ReconMedicalAidFile.fReconMedicalAidID = ReconMedicalAids.fReconMedicalAidID
WHERE (fIsImported = 0) AND (fNumberRecords = 0)
I added this code in the Script Task which will map the selected GUID above:
DELETE FROM BankmedStatments
WHERE fSession = ?
DELETE FROM ReconMedicalAidData
WHERE fSession = ?
Related
I have made the ssis package in which i made the data flow for incremental data. Source and destination server ip's are different. Below you can find the flow diagram of my packageControl flow diagram
Data flow diagram
the package is working fine .
In the Execute SQl task :- it controls the log table and start the incremental task
query which i used is :-
insert into audit_log (
Packagename,
process_date,
start_datetime,
end_datetime,
Record_processed,
status
)values('CRM-TO-TRANSORGDB',null,GETDATE(),null,null,null);
select MAX(ID) as ID,MAX(process_date) as proc_date from audit_log where Packagename ='CRM-TO-TRANSORGDB' ;
store the ID and proc_date in the variable.
in the Execute SQl task 1:- it just update the log table.
UPDATE audit_log
SET
process_date=?,
end_datetime = GETDATE(),
status='SUCCESS'
record_processed=?
WHERE (packagename = 'CRM-TO-TRANSORGDB') AND ID=? ;
this is the query we have used to update the log table.
In the Data flow simple fetching the all the records and put in into the destination table.
this all i have done .
But my question are:-
1) How to compare the total no. of row counts from the source table to destination table in ssis package.
2) if its doesn't matches than it will restart my task automatically.
#thomas as per your instruction i have done the following thing:
1) i have made the Execute SQl Task for source and destination .
2) and Add the Execute Package task and added the condition for not matching the count.
and added the expression for check row_count_src!= row_count_dest
and in Source_table_count i have used the below query:
select count(SubOrderID) as row_count_src from fact_suborder_journey
WHERE Suborderdate between '2016-06-01' and GETDATE()-1 ;
in dest_table_count i have used the below query:
select count(SubOrderID) as row_count_dest from fact_suborder_journey
WHERE Suborderdate between '2016-06-01' and GETDATE()-1 ;
i have added the two variable as int64 in ths ssis package. and map in the result set below you can find the pic what i have done.
but After done all this this i am getting this error:
[Execute SQL Task] Error: An error occurred while assigning a value to variable "row_count_src": "The type of the value being assigned to variable "User::row_count_src" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
".
I havent tested this completely but you might be able to do something like this. This creates a loop of your packages and will executes as long as your count variables are different from each other.
What have i done?
First i have a DataFlow Task which moves data from source to
destination.
Then i have an Execute SQL task which basically counts all rows from
TableA and maps it to variable count1 eg. Source table
Then i have an Execute SQL task which basically counts all rows from
TableB and maps it to variable count2 eg. Destination Table
Then i create an Execute Package task where i reference it too it
self. Then i make a precedence constraint with an expression saying
Count1 != count2.
Because if they are different you want to restart the task. If they
are equal the last task Execute Package task will never be executed.
Hope that is something like that?
If I understand your challenge correctly...
In the data flow task, use a RowCount transformation between source
and destination to capture the rows written to the destination. This
will be stored in a variable.
In the control flow, get the max row counts available from the log table and store that a variable.
Create an execute package tasks that executes this same package and put a precedence constraint before if that compares if variable from Step1 <> variable in Step2.
Does anyone know how I can use RMySQL (or another library) to update a row in a table, rather than having to pull out the full table and push it back in? I don't want to read such a huge table into memory just to update one row.
What I am trying to do is pull out a row, change some of the values in there within R and push the same row object back into the table.
However, dbWriteTable seems to replace the entire table rather than just the row I specify.
The easiest way is to construct a string within R containing the adequate SQL Update statement and use dbSendQuery to push your data back into the table.
Using sqldf package:
library(sqldf)
table_name = data.frame(a = 1:10, b = 4)
# Open connection
sqldf()
fn$sqldf("update table_name set b=1")
ans = sqldf("select * from main.table_name")
# Close connection
sqldf()
print(table_name)
I am trying to execute SELECT ... FOR UPDATE query using Laravel 3:
SELECT * from projects where id = 1 FOR UPDATE;
UPDATE projects SET money = money + 10 where id = 1;
I have tried several things for several hours now:
DB::connection()->pdo->exec($query);
and
DB::query($query)
I have also tried adding START TRANSACTION; ... COMMIT; to the query
and I tried to separate the SELECT from the UPDATE in two different parts like this:
DB::query($select);
DB::query($update);
Sometimes I get 0 rows affected, sometimes I get an error like this one:
SQLSTATE[HY000]: General error: 2014 Cannot execute queries while other unbuffered queries are active. Consider using PDOStatement::fetchAll(). Alternatively, if your code is only ever going to run against mysql, you may enable query buffering by setting the PDO::MYSQL_ATTR_USE_BUFFERED_QUERY attribute.
SQL: UPDATE `sessions` SET `last_activity` = ?, `data` = ? WHERE `id` = ?
I want to lock the row in order to update sensitive data, using Laravel's database connection.
Thanks.
In case all you need to do is increase money by 10, you don't need to lock the row before update. Simply executing the update query will do the job. The SELECT query will only slow down your script and doesn't help in this case.
UPDATE projects SET money = money + 10 where id = 1;
I would use diferent queries for sure, so you can have control on what you are doing.
I would use a transaction.
If we read this simple explanations, pdo transactions are quite straightforward. They give us this simple but complete example, that ilustrates how everithing is as we should expect (consider $db to be your DB::connection()->pdo).
try {
$db->beginTransaction();
$db->exec("SOME QUERY");
$stmt = $db->prepare("SOME OTHER QUERY?");
$stmt->execute(array($value));
$stmt = $db->prepare("YET ANOTHER QUERY??");
$stmt->execute(array($value2, $value3));
$db->commit();
}
catch(PDOException $ex) {
//Something went wrong rollback!
$db->rollBack();
echo $ex->getMessage();
}
Lets go to your real statements. For the first of them, the SELECT ..., i wouldn't use exec, but query, since as stated here
PDO::exec() does not return results from a SELECT statement. For a
SELECT statement that you only need to issue once during your program,
consider issuing PDO::query(). For a statement that you need to issue
multiple times, prepare a PDOStatement object with PDO::prepare() and
issue the statement with PDOStatement::execute().
And assign its result to some temp variable like
$result= $db->query ($select);
After this execution, i would call $result->fetchAll(), or $result->closeCursor(), since as we can read here
If you do not fetch all of the data in a result set before issuing
your next call to PDO::query(), your call may fail. Call
PDOStatement::closeCursor() to release the database resources
associated with the PDOStatement object before issuing your next call
to PDO::query().
Then you can exec the update
$result= $db->exec($update);
And after all, just in case, i would call again $result->fetchAll(), or $result->closeCursor().
If the aim is
to lock the row in order to update sensitive data, using Laravel's database connection.
Maybe you can use PDO transactions :
DB::connection()->pdo->beginTransaction();
DB::connection()->pdo->commit();
DB::connection()->pdo->rollBack();
This one is a pretty good one IMO and I have not seen a close exampled on SO or Google so here you go. I need to do the following within a Perl application I am building. Unfortunately it can not be done directly in MySQL and will require DBI. In a nutshell I need to take Database1.tableA and locate every record with the column 'status' matching 'started'. This I can do as it is fairly easy (not very good with DBI yet, but have read the docs), but where I am having issues is what I have to do next.
my $started_query = "SELECT primary_ip FROM queue WHERE status='started'";
my $started = $dbh->prepare($started_query);
$started->execute();
while ( my #started = $started->fetchrow_array() ) {
# Where I am hoping to have the following occur so it can go by row
# for only rows with the status 'started'
}
So for each record in the #started array, really only contains one value per iteration of the while loop, I need to see if it exists in the Database2.tableA and IF it does exist in the other database (Database2.tableA) I need to delete it from Database1.tableA, but if it DOES NOT exist in the other database (Database2.tableA) I need to update the record in the current database (Database1.tableA).
Basically replicating the below semi-valid MySQL syntax.
DELETE FROM tableA WHERE primary_ip IN (SELECT primary_ip FROM db2.tablea) OR UPDATE tableA SET status = 'error'
I am limited to DBI to connect to the two databases and the logic is escaping me currently. I could do the queries to both databases and store in #arrays and then do a comparison, but that seems redundant as I think it should be possible within the while ( my #started = $started->fetchrow_array() ) as that will save on runtime and resources required. I am also not familiar enough with passing variables between DBI instances and as the #started array will always contain the column value I need to query for and delete I would like to take full advantage of having that defined and passed to the DBI objects.
I am going to be working on this thing all night and already ran through a couple pots of coffee so your helping me understand this logic is greatly appreciated.
You'll be better off with fetchrow_hashref, which returns a hashref of key/value pairs, where the keys are the column names, rather than coding based on columns showing up at ordinal positions in the array.
You need an additional database handle to do the lookups and updates because you've got a live statement handle on the first one. Something like this:
my $dbh2 = DBI->connect(...same credentials...);
...
while(my $row = $started->fetchrow_hashref)
{
if(my $found = $dbh2->selectrow_hashref("SELECT * FROM db2.t2 WHERE primary_ip = ?",undef,$row->{primary_ip}))
{
$dbh2->do("DELETE FROM db1.t1 WHERE primary_ip = ?",undef,$found->{primary_ip});
}
else
{
$dbh2->do("UPDATE db1.t1 SET status = 'error' WHERE primary_ip = ?",undef,$found->{primary_ip}");
}
}
Technically, I don't "need" to fetch the row from db2.t2 into my $found since you're only apparently testing for existence, there are other ways, but using it here is a bit of insurance against doing something other than you intended, since it will be undef if we somehow get some bad logic going and that should keep us from making some potential wrong changes.
But approaching a relational database with loop iterations is rarely the best tactic.
This "could" be done directly in MySQL with just a couple of queries.
First, the updates, where t1.status = 'started' and t2.primary_ip has no matching value for t1.primary_ip:
UPDATE db1.t1 a LEFT JOIN db2.t2 b ON b.primary_ip = a.primary_ip
SET a.status = 'error'
WHERE b.primary_ip IS NULL AND a.status = 'started';
If you are thinking "but b.primary_ip is never null" ... well, it is null in a left join where there are no matching rows.
Then deleting the rows from t1 can also be accomplished with a join. Multi-table joins delete only the rows from the table aliases listed between DELETE and FROM. Again, we're calling "t1" by the alias "a" and t2 by the alias "b".
DELETE a
FROM db1.t1 a JOIN db2.t2 b ON a.primary_ip = b.primary_ip
WHERE a.status = 'started';
This removes every row from t1 ("a") where status = 'started' AND where a matching row exists in t2.
I have separated Matching rows with Lookup and stored the matched data into a temp table through ODLEDB Destination.
After finishing data flow task its getting error on Execute SQL Task.
Property was set as
ResultSet: "NONE";
ConnectionType:"OLEDB";
Connection:"myDatabase";
SQLType: "Direct Input"
SQLStatement
Update dbo.[Payment_Transaction]
SET pt.[Date] = tt.[Date], pt.[FromCurrency]=tt.FromCurrency], pt.[ToCurrency]=tt.[ToCurrency], pt.[TransRate]=tt.[TransRate], pt.[TransType] =tt.[TransType], pt.[TransAmount]=tt.[TransAmount]
FROM dbo.[Payment_Transaction] pt INNER JOIN [##temp_table] tt ON pt.[ID] = tt.[ID]
Getting ERROR saying
"Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly".
Can anyone please shade any light where it could be wring?
Thanks,
Jahid
Hi I guess is tmp table related so...
try with "Set RetainSameConnection=TRUE on the connection manager and set DelayValidation=TRUE on all tasks that use the temp table. Setting RetainSameConnection=TRUE ensures that the temp table does not disappear when the task that creates it has completed. Setting DelayValidation=TRUE ensures that the tasks do not try to check that the table exists before it is created."
http://consultingblogs.emc.com/jamiethomson/archive/2006/11/19/SSIS_3A00_-Using-temporary-tables.aspx
Cheers, Mario