select
FiscalMonthID = (select FiscalMonthID from CurrentFiscalMonth (nolock)),
T.OrgKey,
DataSourceKey = 26,
OrganizationTypeKey = 2,
SourceSystemID = MAX(T.MbsId),
WEGFlag = convert(bit,0),
D.CreateDT,
D.CreateBy,
D.UpdateDT,
D.UpdateBy
from WorkDB.dbo.TempMbsOrgMap (nolock) as T
join WorkDB.dbo.MBSOrganization_Denorm2 (nolock) as D
on T.MbsId = D.OrganizationID
--where OrgKey not in (select OrgKey from OrgMap where FiscalMonthID=258 and DataSourceKey=26 and OrganizationTypeKey=2)
group by
T.OrgKey,
D.CreateDT,
D.CreateBy,
D.UpdateDT,
D.UpdateBy
I don't know whether this person ever got the problem resolved. If anyone else hits this error, the most helpful article I have found so far is:
https://learn.microsoft.com/en-us/archive/blogs/sqlserverfaq/executing-bcp-fails-with-sqlstate-37000-nativeerror-4891-error-microsoftodbc-sql-server-driversql-serverinsert-bulk-failed-due-to-a-schema-change-of-the-target-table
The recommendations from the article (it's worth reading the whole thing though) are as follows:
Below is some of the action plan you can try, this helped in my case
though.
Drop the Constraints before the BCP run and recreate them after the
run
Disable the Auto update stats (To isolate the issue)
Check if any parallel index rebuilds happening.
If still the issue persists after implementing the above change,
collect the Profiler trace to capture the activity when bcp is failing
to further investigation.
Related
I have several procedures with same name but in different schemas. When these procedures raise an error, it is possible in parent procedure (which is calling these nested stored procedures) get schema of the procedure which raised an error ? For example i can get name from ERROR_PROCEDURE() but is there some option to get also SCHEMA ? Because otherwise i am not sure which exactly procedure throwed an error if there are many with same name.
I guess this feature is still missing
https://connect.microsoft.com/SQLServer/feedback/details/124627/schema-not-reported-in-the-error-procedure-function
but is there some workaround for this ?
Sadly, there is no 100% Workaround for this limitation in SQL-Server.
Shame on the MSSQL Dev Team for not rectifying this, well over a Decade later.
It should be as simple as adding a New Function like ERROR_ProcedureSchema() or ERROR_PROCID().
Here is a revived Post Requesting this Feature from way back in May of 2005:
https://feedback.azure.com/forums/908035-sql-server/suggestions/32894584-schema-not-reported-in-the-error-procedure-functio
I prefer to Log as much detail as possible about Exceptions I capture in my custom Error Handling Logic.
This is the best I could come up with to find the Schema Name:
DECLARE #Error_ProcSchemaName nVarChar(128)--Leave as Null if found in more than 1 Schema.
--Only Populate the #Error_ProcSchemaName if it Belongs to 1 Schema. - 04/08/2019 - MCR.
SELECT #Error_ProcSchemaName = S.name
FROM sys.objects as O
JOIN sys.schemas as S
ON S.schema_id = O.schema_id
JOIN
(
SELECT O.name[ObjectName], COUNT(*)[Occurrences]
FROM sys.objects as O
GROUP BY O.name
) AS Total
ON Total.ObjectName = O.name
WHERE O.name = ERROR_PROCEDURE()
AND Total.Occurrences = 1
Avoid using anything like OBJECT_SCHEMA_NAME(OBJECT_ID(ERROR_PROCEDURE())) as the string you pass into OBJECT_ID() should already have the Schema in it (which ERROR_PROCEDURE() does not).
Otherwise it will default to your Default Schema, which (in most cases) is dbo.
Run this Query to View all your Object Names that are Reused across Schemas:
--View Object Names that Exist in Multiple Schemas: - 04/08/2019 - MCR.
SELECT S.name[SchemaName], O.name[ObjectName], Total.Occurrences,
O.type[Type], O.type_desc[TypeDesc],
O.object_id[ObjectID], O.principal_id[PrincipalID], O.parent_object_id[ParentID],
O.is_ms_shipped[MS], O.create_date[Created], O.modify_date[Modified]
FROM sys.objects as O
JOIN sys.schemas as S
ON S.schema_id = O.schema_id
JOIN
(
SELECT O.name[ObjectName], COUNT(*)[Occurrences]
FROM sys.objects as O
GROUP BY O.name
) AS Total
ON Total.ObjectName = O.name
WHERE Total.Occurrences > 1
ORDER BY [ObjectName], [SchemaName]
If you only have a few Objects (Sprocs and Triggers) that overlap, then you might be okay not knowing the Schema as it may be obvious where it originated from.
However, if this is not the case, then you may need to either:
Change the Name of the Sproc/Trigger to make it Unique.
This option goes against the very fiber of my being.
If you are using Advanced Error Handling, then manually add the Schema of your Sproc/Trigger with OBJECT_SCHEMA_NAME(##PROCID)
in your Catch-Block when logging the Error.
Note: These options may not be possible due to the use of 3rd Party Sprocs you are not allowed to edit.
When Troubleshooting with multiple Sprocs/Triggers that share the same Name, you might be able to write a Custom Wrapper-Sproc to call your 3rd Party Sproc, then Log any exception thrown in your Wrapper to know exactly which Schema/Sproc caused it.
The Code Smell:
If you have multiple Sprocs/Triggers with the same name spread across various Schemas
then I would call that a "Code Smell".
Meaning, your Architecture is flawed.
You may not be properly encapsulating your logic for reuse.
There will be times when a name overlaps Schemas, but this should be rare and by coincidence only.
Misappropriating Schemas for Handling Multi-Tenant / UserGroup Access:
If you are trying something Multi-Tenant (storing data from different Organizations/UserGroups in the same Database and preventing them from seeing eachother's info) and running almost the same logic in each Schema that shares the Object Name, then that's a Design Problem.
You should have your data in Different Databases if Users will be accessing it directly
or have a TenantID or UserGroupID you always pass in and filter on everywhere when Users will be accessing from a Custom Application.
Some possible solutions I can think of:
Renaming each Stored Procedure so that they have different names in the different schemas.
Adding some debugging output to the Stored Procedures so that when they are being executed, you can see which one was in progress when your error occurred.
Running the SQL Profiler to see what is being called at the time your error occurs.
However, these are coming more from the perspective of trying to troubleshoot an issue you're having right now, rather than building in some error handling for potential future troubleshooting. You could always get these Stored Procedures to write some log files to disk somewhere so you can interrogate those logs when an error is experienced perhaps.
I am currently writing a web app (Flask) using SQLAlchemy (on GAE, connecting to Google's cloud MySQL) and needing to do bulk updates of a table. In short, a number of calculations are done resulting in a single value needing to be updated on 1000's of objects. At the moment I'm doing it all in a transaction, but still at the end, the flush/commit is taking ages.
The table has an index on id and this is all carried out in a single transaction. So I believe I've avoided the usual mistakes, but is is still very slow.
INFO 2017-01-26 00:45:46,412 log.py:109] UPDATE wallet SET balance=%(balance)s WHERE wallet.id = %(wallet_id)s
2017-01-26 00:45:46,418 INFO sqlalchemy.engine.base.Engine ({'wallet_id': u'3c291a05-e2ed-11e6-9b55-19626d8c7624', 'balance': 1.8711760000000002}, {'wallet_id': u'3c352035-e2ed-11e6-a64c-19626d8c7624', 'balance': 1.5875759999999999}, {'wallet_id': u'3c52c047-e2ed-11e6-a903-19626d8c7624', 'balance': 1.441656}
From my understanding there is no way to do a bulk update in SQL actually, and the statement above ends up being multiple UPDATE statements being sent to the server.
I've tried using Session.bulk_update_mappings() but that doesn't seem to actually do anything :( Not sure why, but the updates never actually happen. I can't see any examples of this method actually being used (including in the performance suite) so not sure if it is intended to be used.
One technique I've seen discussed is doing a bulk insert into another table and then doing an UPDATE JOIN. I've given it a test, like below, and it seems to be significantly faster.
wallets = db_session.query(Wallet).all()
ledgers = [ Ledger(id=w.id, amount=w._balance) for w in wallets ]
db_session.bulk_save_objects(ledgers)
db_session.execute('UPDATE wallet w JOIN ledger l on w.id = l.id SET w.balance = l.amount')
db_session.execute('TRUNCATE ledger')
But the problem now is how to structure my code. I'm using the ORM and I need to somehow not 'dirty' the original Wallet objects so that they don't get committed in the old way. I could just create these Ledger objects instead and keep a list of them about and then manually insert them at the end of my bulk operation. But that almost smells like I'm replicating some of the work of the ORM mechanism.
Is there a smarter way to do this? So far my brain is going down something like:
class Wallet(Base):
...
_balance = Column(Float)
...
#property
def balance(self):
# first check if we have a ledger of the same id
# and return the amount in that, otherwise...
return self._balance
#balance.setter
def balance(self, amount):
l = Ledger(id=self.id, amount=amount)
# add l to a list somewhere then process later
# At the end of the transaction, do a bulk insert of Ledgers
# and then do an UPDATE JOIN and TRUNCATE
As I said, this all seems to be fighting against the tools I (may) have. Is there a better way to be handling this? Can I tap into the ORM mechanism to be doing this? Or is there an even better way to do the bulk updates?
EDIT: Or is there maybe something clever with events and sessions? Maybe before_flush?
EDIT 2: So I have tried to tap into the event machinery and now have this:
#event.listens_for(SignallingSession, 'before_flush')
def before_flush(session, flush_context, instances):
ledgers = []
if session.dirty:
for elem in session.dirty:
if ( session.is_modified(elem, include_collections=False) ):
if isinstance(elem, Wallet):
session.expunge(elem)
ledgers.append(Ledger(id=elem.id, amount=elem.balance))
if ledgers:
session.bulk_save_objects(ledgers)
session.execute('UPDATE wallet w JOIN ledger l on w.id = l.id SET w.balance = l.amount')
session.execute('TRUNCATE ledger')
Which seems pretty hacky and evil to me, but appears to work OK. Any pitfalls, or better approaches?
-Matt
What you're essentially doing is bypassing the ORM in order to optimize the performance. Therefore, don't be surprised that you're "replicating the work the ORM is doing" because that's exactly what you need to do.
Unless you have a lot of places where you need to do bulk updates like this, I would recommend against the magical event approach; simply writing the explicit queries is much more straightforward.
What I recommend doing is using SQLAlchemy Core instead of the ORM to do the update:
ledger = Table("ledger", db.metadata,
Column("wallet_id", Integer, primary_key=True),
Column("new_balance", Float),
prefixes=["TEMPORARY"],
)
wallets = db_session.query(Wallet).all()
# figure out new balances
balance_map = {}
for w in wallets:
balance_map[w.id] = calculate_new_balance(w)
# create temp table with balances we need to update
ledger.create(bind=db.session.get_bind())
# insert update data
db.session.execute(ledger.insert().values([{"wallet_id": k, "new_balance": v}
for k, v in balance_map.items()])
# perform update
db.session.execute(Wallet.__table__
.update()
.values(balance=ledger.c.new_balance)
.where(Wallet.__table__.c.id == ledger.c.wallet_id))
# drop temp table
ledger.drop(bind=db.session.get_bind())
# commit changes
db.session.commit()
Generally it is poor schema design to need to update thousands of rows frequently. That aside...
Plan A: Write ORM code that generates
START TRANSACTION;
UPDATE wallet SET balance = ... WHERE id = ...;
UPDATE wallet SET balance = ... WHERE id = ...;
UPDATE wallet SET balance = ... WHERE id = ...;
...
COMMIT;
Plan B: Write ORM code that generates
CREATE TEMPORARY TABLE ToDo (
id ...,
new_balance ...
);
INSERT INTO ToDo -- either one row at a time, or a bulk insert
UPDATE wallet
JOIN ToDo USING(id)
SET wallet.balance = ToDo.new_balance; -- bulk update
(Check the syntax; test; etc.)
Started to get following error when executing certain SP. Code related to this error is pretty simple, joining #temp table to real table
Full text of error:
Msg 605, Level 21, State 3, Procedure spSSRSRPTIncorrectRevenue, Line 123
Attempt to fetch logical page (1:558552) in database 2 failed. It belongs to allocation unit 2089673263876079616 not to 4179358581172469760.
Here is what I found:
https://support.microsoft.com/en-us/kb/2015739
This suggests some kind of issue with database. I run DBCC CHECKDB on user database and on temp database - all passes.
Second thing I'm doing - trying to find which table those allocation units belong
SELECT au.allocation_unit_id, OBJECT_NAME(p.object_id) AS table_name, fg.name AS filegroup_name,
au.type_desc AS allocation_type, au.data_pages, partition_number
FROM sys.allocation_units AS au
JOIN sys.partitions AS p ON au.container_id = p.partition_id
JOIN sys.filegroups AS fg ON fg.data_space_id = au.data_space_id
WHERE au.allocation_unit_id in(2089673263876079616, 4179358581172469760)
ORDER BY au.allocation_unit_id
This returns 2 objects in tempdb, not in user db. So, it makes me think it's some kind of data corruption in tempdb? I'm developer, not DBA. Any suggestions on what I should check next?
Also, when I run query above, how can I tell REAL object name that I understand? Like #myTempTable______... instead of #07C650CE
I was able to resolve this by clearing the SQL caches:
DBCC FREEPROCCACHE
GO
DBCC DROPCLEANBUFFERS
GO
Apparently restarting the SQL service would have had the same affect.
(via Made By SQL, reproduced here to help others!)
I have like your get errors too.
firstly you must backing up to table or object for dont panic more after. I tryed below steps on my Database.
step 1:
Backing up table (data movement to other table as manuel or vs..how can you do)
I used to below codes to my table move other table
--CODE-
set nocount on;
DECLARE #Counter INT = 1;
DECLARE #LastRecord INT = 10000000; --your table_count
WHILE #Counter < #LastRecord
BEGIN
BEGIN TRY
BEGIN
insert into your_table_new SELECT * FROM your_table WHERE your_column= #Counter --dont forget! create your_table_new before
END
END TRY
BEGIN CATCH
BEGIN
insert into error_code select #Counter,'error_number' --dont forget the create error_code table before.
END
END CATCH
SET #Counter += 1;
END;
step 2:
-DBCC CHECKTABLE(your_table , REPAIR_REBUILD )
GO
check your table. if you have an error go to other step_3.
step 3:
!!attention!! you can lost some data/datas on your table. but dont worry. so you backed-up your table in step_1.
-DBCC CHECKTABLE(your_table , REPAIR_ALLOW_DATA_LOSS)
GO
Good luck!
~~pektas
In my case, truncating and re-populating data in the concerned tables was the solution.
Most probably the data inside tables was corrupted.
Database ID 2 means your tempdb is corrupted. Fixing tempdp is easy. Restart sqlserver service and you are good to go.
This could be an instance of a bug Microsoft fixed on SQL Server 2008 with queries on temporary tables that self reference (for example we have experienced it when loading data from a real table to a temporary table while filtering any rows we already have populated in the temp table in a previous step).
It seems that it only happens on temporary tables with no identity/primary key, so a workaround is to add one, although if you patch CU3 or later you also can enable the hotfix via turning a trace flag on.
For more details on the bug/fixes: https://support.microsoft.com/en-us/help/960770/fix-you-receive-error-605-and-error-824-when-you-run-a-query-that-inse
When I execute this code in the developer console
PushTopic pushTopic = new PushTopic();
pushTopic.ApiVersion = 23.0;
pushTopic.Name = 'Test';
pushTopic.Description = 'test';
pushtopic.Query = 'SELECT Id, Account.Name FROM Case';
insert pushTopic;
System.debug('Created new PushTopic: '+ pushTopic.Id);
I receive this message:
FATAL ERROR System.DmlException: Insert failed. First exception on row
0; first error: INVALID_FIELD, relationships are not supported:
[QUERY]
The same query runs fine on the Query Editor, but when I assign it to a Push Topic I get the INVALID_FIELD exception.
If the bottom line is what the exception message says, that relationships are just not supported by Push Topic objects, how do I create a Push Topic object that will return the data I'm looking for?
Why
Salesforce prevents this because it will require them to join tables, joins in salesforces database are expensive due to the multi-tenancy. Usually when they add a new feature they will not support joins as it requires more optimization of the feature.
Push Topics are still quite new to the system and need to be real time, anything that would slow them down I'd say needs to be trimmed.
I'd suggest you look more closely at your requirement and see if there is something else that will work for you.
Workaround
A potential workaround is to add a Formula field to the Case object with the data you need and include that in the query instead. This may not work as it will still require a join to work.
A final option may be to use a workflow rule or trigger to update the account name to a custom field on the Case object this way the data is local so doesn't require a join...
PushTopics support a very small subset of SOQL queries, see more here:
https://developer.salesforce.com/docs/atlas.en-us.api_streaming.meta/api_streaming/unsupported_soql_statements.htm
However this should work:
PushTopic casePushTopic = new PushTopic();
pushTopic.ApiVersion = 23.0;
pushTopic.Name = 'CaseTopic';
pushTopic.Description = 'test';
pushtopic.Query = 'SELECT Id, Account.Id FROM Case';
insert pushTopic;
PushTopic accountPushTopic = new PushTopic();
pushTopic.ApiVersion = 23.0;
pushTopic.Name = 'AccountTopic';
pushTopic.Description = 'test';
pushtopic.Query = 'SELECT Id, Name FROM Account';
insert pushTopic;
It really depends on your use case though, if it is for replicating into RDBMS this should be enough, you can use a join to get the full data.
I have a table that is storing data that needs to be processed. I have id, status, data in the table. I'm currently going through and selecting id, data where status = #. I'm then doing an update immediately after the select, changing the status # so that it won't be selected again.
my program is multithreaded and sometimes I get threads that grab the same id as they are both querying the table at a relatively close time to each other, causing the grab of the same id. i looked into select for update, however, i either did the query wrong, or i'm not understanding what it is used for.
my goal is to find a way of grabbing the id, data that i need and setting the status so that no other thread tries to grab and process the same data. here is the code i tried. (i wrote it all together for show purpose here. i have my prepares set at the beginning of the program as to not do a prepare for each time it's ran, just in case anyone was concerned there)
my $select = $db->prepare("SELECT id, data FROM `TestTable` WHERE _status=4 LIMIT ? FOR UPDATE") or die $DBI::errstr;
if ($select->execute($limit))
{
while ($data = $select->fetchrow_hashref())
{
my $update_status = $db->prepare( "UPDATE `TestTable` SET _status = ?, data = ? WHERE _id=?");
$update_status->execute(10, "", $data->{_id});
push(#array_hash, $data);
}
}
when i run this, if doing multiple threads, i'll get many duplicate inserts, when trying to do an insert after i process my transaction data.
i'm not terribly familiar with mysql and the research i've done, i haven't found anything that really cleared this up for me.
thanks
As a sanity check, are you using InnoDB? MyISAM has zero transactional support, aside from faking it with full table locking.
I don't see where you're starting a transaction. MySQL's autocommit option is on by default, so starting a transaction and later committing would be necessary unless you turned off autocommit.
It looks like you simply rely on the database locking mechanisms. I googled perl dbi locking and found this:
$dbh->do("LOCK TABLES foo WRITE, bar READ");
$sth->prepare("SELECT x,y,z FROM bar");
$sth2->prepare("INSERT INTO foo SET a = ?");
while (#ary = $sth->fetchrow_array()) {
$sth2->$execute($ary[0]);
}
$sth2->finish();
$sth->finish();
$dbh->do("UNLOCK TABLES");
Not really saying GIYF as I am also fairly novice at both MySQL and DBI, but perhaps you can find other answers that way.
Another option might be as follows, and this only works if you control all the code accessing the data. You can create lock column in the table. When your code accesses the table it (pseudocode):
if row.lock != 1
row.lock = 1
read row
update row
row.lock = 0
next
else
sleep 1
redo
again though, this trusts that all users/script that access this data will agree to follow this policy. If you cannot ensure that then this won't work.
Anyway thats all the knowledge I have on the topic. Good Luck!