I'm trying to fit Linq to Sql into an N-Tier design. I am implementing concurrency by supplying original values when attaching objects to the data-context. When calling SubmitChanges, and observing the generated scripts on sql server profiler, I can see that they are being generated properly. They include where clauses that check all the object properties (they are all marked with UpdateCheck.Always).
The result is as expected, i.e., no rows are updated on updates or deleted on deletes. Yet I am not getting any exception. Isn't this supposed to throw a ChangeConflictException?
For clarity here is the design and flow for the tests I'm running: I have a client console and a service console talking to each other via WCF using WsHttpBinding.
Client requests data from service
Service instantiates a datacontext, retrieves data, disposes context, returns data to client.
Client makes modifications to returned data.
Client requests an update of changed data from the service.
5a. Service instantiates a datacontext, attaches objects, and...
5b. I pause execution and change values in the database in order to cause a change-conflict
5c. Service calls SubmitChanges.
Here's the code for step 5, cleaned up a bit for clarity:
public void UpdateEntities(ReadOnlyChangeSet<Entity> changeSet)
{
using (EntityDataContext context = new EntityDataContext())
{
if (changeSet.AddedEntities.Count > 0)
{
context.Entities.InsertAllOnSubmit(changeSet.AddedEntities);
}
if (changeSet.RemovedEntities.Count > 0)
{
context.Entities.AttachAll(changeSet.RemovedEntities, false);
context.Entities.DeleteAllOnSubmit(changeSet.RemovedEntities);
}
if (changeSet.ModifiedRecords.Count > 0)
{
foreach (var record in changeSet.ModifiedRecords)
{
context.Entities.Attach(record.Current, record.Original);
}
}
// This is where I pause execution and make changes to the database
context.SubmitChanges();
}
}
I'm using some classes to track changes and maintain originals, as you can see.
Any help appreciated.
EDIT: I'm having no problems with inserts. I've only included the code that calls InsertAllOnSubmit for completeness.
So I've found the answer. It appears to be a bug in Linq To Sql (correct me if I'm wrong). It turns out that the table being updated in the database has a trigger on it. This trigger calls a stored procedure that has a return value. This causes calls to insert, update or delete on this table to yield a return value (from the stored procedure run by the trigger) which is NOT a row-count but is a number. Apparently L2S sees this number and assumes all went well even though no insert/update/delete actually occurred.
This is quite bizarre, especially considering the returned number has a defined column name and its value is in the 6-digit area.
Related
Can anyone advise on the following problem:
I have a custom get_or_create method, which checks multiple fields and does some fancy stuff upon creation:
def fancy_get_or_create(name):
object = self.fancy_get(name)
if not object:
object = self.fancy_create(name)
return object
def fancy_get(name):
return self.filter(Q(name=name) | Q(alias=name)).first()
def fancy_create(name):
name = self.some_preprocessing(name)
return self.create(name=name, alias=name)
There's a race condition, where one request will check to see if the object exists, find nothing, and start creating it. Before that request finishes creating the object, another request comes in looking for the same object, finds, nothing, and begins creating the new object. This request will fail because the database has some uniqueness constraints (the previous request had just created the object, so the second request will fail).
Is there any way to prevent request 2 from querying the database until request 1 has finished? I was reading about transaction management and it did not seem like the solution, since the issue is not partial updates (which would suggest an atomic transaction), but rather the need to make the second request wait until the first has finished.
Thanks!
Update:
Here's what I went with:
try:
return self.fancy_get(name) or self.fancy_create(name)
except IntegrityError:
return self.fancy_get(name)
There are two viable solutions:
Use a mutex so only one process can access the fancy_get_or_create
function at the same time.
Capture the error thrown by the database and do something instead: ignore
that create, update the row instead of creating it, throw an
exception, etc.
Edit: another solution might be doing an INSERT IGNORE instead of just an INSERT. https://dev.mysql.com/doc/refman/5.1/en/insert.html
I have a view ObjectDisplay that is composed of two relevant tables: Object and State. State represents the state of an Object, and the view pulls some of the details from the most recent State for each Object.
On the page that is displaying this information, a user can enter some comments, which creates a new State. After creating the new State, I immediately pull the Object from ObjectDisplay and send it back to be dropped into a partial view and replace the Object in the grid on the page.
// Add new State.
db.States.Add(new State()
{
ObjectId = objectId,
Comments = comments,
UserName = username
});
// Save the changes (executes all of the above).
db.SaveChanges();
// Return the new Object information.
return db.Objects.Single(c => c.ObjectId == objectId);
According to my db trace, the Single call occurs about 70 ms after the SaveChanges call, and it occurs on the same SPID.
Now for the issue: The database defaults the value of RecordDate in State to GETUTCDATE() - I don't provide the date myself. What I'm seeing is that the Object returned has the State's RecordDate of the old State and the Comments of the new State information of the old State. I am seeing that the Object returned has the old State's information. When I refresh the page, all the correct information is there, but the wrong information is returned in the initial call from the database/EF.
So.. what could be wrong? Could the view not be updating quickly enough? Could something be going on with EF? I don't really know where to start looking.
If you've previously loaded the same Object entity in the same DbContext, EF will return the cached instance with the stale values, and ignore the values returned from SQL.
The simplest solution is to reload the entity before returning it:
var result = db.Objects.Single(c => c.ObjectId == objectId);
db.Entry(result).Reload();
return result;
This is indeed odd. In SQL Server views are not persisted by default and therefore show changes in the underlying data right away. You can create a clustered index on a view with effectively persists the query, but in that case the data is updated synchronously, so you should see the change right away.
If you are working with snapshot isolation level your changes might not be visible to other SPIDs right away, but as you are on the same SPID and do not use snapshot isolation, this cant be the culprit either.
The only thing left at this point is the application layer. Are you actually using the result of the Single call higher up in the call stack or does that get lost somewhere. I assume that a refresh of the page uses a different code path, which would explain why it is working there.
is there a way to call the reindexing that you can fire off in magento's backend though an SQL statement?
i have a bunch of scripts which add products to magento and we need to reindex after it, we have a scheduled job that runs these scripts and i want to do the reindex after their done so that way it will always reindex after the scripts are done regardless how long they take (sometimes they can take a couple of minutes, sometimes half and hour, depending on what data needs to be changed, inserted or deleted)
the task scheduler is on a Microsoft SQL Server and magento is on a MySQL server (we use a link server apparently)
No, there is not.
In Magento "re-indexing" means "run through a list of PHP classes and run their reindexAll methods". Indexing strategy varies between indexer types. Most require reading some sort of data, doing programatic calculations, and then inserting values into flat tables.
For example, the catalog/URL rewrite re-indexer is the class
app/code/core/Mage/Catalog/Model/Indexer/Url.php
(alias of catalog/indexer_url, PHP class of Mage_Catalog_Model_Indexer_Url)
Its reindxAll method contains
public function reindexAll()
{
/** #var $resourceModel Mage_Catalog_Model_Resource_Url */
$resourceModel = Mage::getResourceSingleton('catalog/url');
$resourceModel->beginTransaction();
try {
Mage::getSingleton('catalog/url')->refreshRewrites();
$resourceModel->commit();
} catch (Exception $e) {
$resourceModel->rollBack();
throw $e;
}
}
And the actual indexing is handled in the refreshRewrites method, which creates the needed Magento rewrites.
First, I will try to describe what I am willing to do and then, I will ask my questions.
I need to do the following:
List all rows corresponding to some conditions
Do some tests (e.g: check if it wasn't already inserted), if test passes then insert row into another database
Delete row (whether it passed tests or not)
The following is my implementation
List<MyObject> toAdd = new ArrayList<MyObject>();
for(MyObject obj:list){
if(!notYetInserted(obj){
toAdd.add(obj);
}
}
myObjectDAO.insertList(toAdd);
myObjectDAO.deleteList(list);
The service method is marked transactional.
In my DAO methods for deleteList and insertList are pretty similar so I will just put here method for insert.
public void insertList(final List<MyObject> list){
String sql = "INSERT INTO table_test " +
"(col_id, col2, col3,col4) VALUES (?, ?, ?,?)";
List<Object[]> paramList = new ArrayList<Object[]>();
for (MyObject myObject : list) {
paramList.add(new Object[] {myObject.getColId(),
myObject.getCol2(), myObject .getCol3(), myObject.getCol4()}
);
}
simpleJdbcTemplate.batchUpdate(sql, paramList);
}
I am not sure about the best way to perform such operations, I read here that calling for update inside a loop may slow down the system (especially in my case, I will have about 100K insert/delete at a time). I wonder if these additional loops inside DAO won't slow down my system even more and what would happen if problem happened repeatedly while processing that batch (I thought also about moving test from service to DAO to have only one loop and an additional test, I don't really know if it's a good idea). So, I would like your advices. Thanks a lot.
PS: if you need more details feel free to ask!
This is not necessarily a bad approach, but you are right, it might be really slow. If I were to do a process like this that inserted or deleted this many rows I would probably do it all in a stored procedure. My code would just execute the proc and the proc would handle the list and the enumeration through it (as well as the inserts and deletes).
I have some tables in a MySQL database to represent records from a sensor. One of the features of the system I'm developing is to display this records from the database to the web user, so I used ADO.NET Entity Data Model to create an ORM, used Linq to SQL to get the data from the database, and stored them in a ViewModel I designed, so I can display it using MVCContrib Grid Helper:
public IQueryable<TrendSignalRecord> GetTrends()
{
var dataContext = new SmgerEntities();
var trendSignalRecords = from e in dataContext.TrendSignalRecords
select e;
return trendSignalRecords;
}
public IQueryable<TrendRecordViewModel> GetTrendsProjected()
{
var projectedTrendRecords = from t in GetTrends()
select new TrendRecordViewModel
{
TrendID = t.ID,
TrendName = t.TrendSignalSetting.Name,
GeneratingUnitID = t.TrendSignalSetting.TrendSetting.GeneratingUnit_ID,
//{...}
Unit = t.TrendSignalSetting.Unit
};
return projectedTrendRecords;
}
I call the GetTrendsProjectedMethod and then I use Linq to SQL to select only the records I want. It is working fine in my developing scenario, but when I test it in a real scenario, where the number of records is way greater (something around a million records), it stops working.
I put some debug messages to test it, and everything works fine, but when it reaches the return View() statement, it simply stops, throwing me a MySQLException: Timeout expired. That let me wondering if the data I sent to the page is retrieved by the page itself (it only search for the displayed items in the database when the page itself needs it, or something like that).
All of my other pages use the same set of tools: MVCContrib Grid Helper, ADO.NET, Linq to SQL, MySQL, and everything else works alright.
You absolutely should paginate your data set before executing your query if you have millions of records. This could be done using the .Skip and .Take extension methods. And those should be called before running any query against your database.
Trying to fetch millions of records from a database without pagination would very likely cause a timeout at best.
Well, assuming information in this blog is correct, .AsPagination method requires you to sort your data by a particular column. It's possible that trying to do an OrderBy on a table with millions of records in it is just a time consuming operation and times out.