Linq to SQL concurrency problem - linq-to-sql

Hallo,
I have web service that has multiple methods that can be called. Each time one of these methods is called I am logging the call to a statistics database so we know how many times each method is called each month and the average process time.
Each time I log statistic data I first check the database to see if that method for the current month already exists, if not the row is created and added. If it already exists I update the needed columns to the database.
My problem is that sometimes when I update a row I get the "Row not found or changed" exception and yes I know it is because the row has been modified since I read it.
To solve this I have tried using the following without success:
Use using around my datacontext.
Use using around a TransactionScope.
Use a mutex, this doesn’t work because the web service is (not sure I am calling it the right think) replicated out on different PC for performance but still using the same database.
Resolve concurrency conflict in the exception, this doesn’t work because I need to get the new database value and add a value to it.
Below I have added the code used to log the statistics data. Any help would be appreciated very much.
public class StatisticsGateway : IStatisticsGateway
{
#region member variables
private StatisticsDataContext db;
#endregion
#region Singleton
[ThreadStatic]
private static IStatisticsGateway instance;
[ThreadStatic]
private static DateTime lastEntryTime = DateTime.MinValue;
public static IStatisticsGateway Instance
{
get
{
if (!lastEntryTime.Equals(OperationState.EntryTime) || instance == null)
{
instance = new StatisticsGateway();
lastEntryTime = OperationState.EntryTime;
}
return instance;
}
}
#endregion
#region constructor / initialize
private StatisticsGateway()
{
var configurationAppSettings = new System.Configuration.AppSettingsReader();
var connectionString = ((string)(configurationAppSettings.GetValue("sqlConnection1.ConnectionString", typeof(string))));
db = new StatisticsDataContext(connectionString);
}
#endregion
#region IStatisticsGateway members
public void AddStatisticRecord(StatisticRecord record)
{
using (db)
{
var existing = db.Statistics.SingleOrDefault(p => p.MethodName == record.MethodName &&
p.CountryID == record.CountryID &&
p.TokenType == record.TokenType &&
p.Year == record.Year &&
p.Month == record.Month);
if (existing == null)
{
//Add new row
this.AddNewRecord(record);
return;
}
//Update
existing.Count += record.Count;
existing.TotalTimeValue += record.TotalTimeValue;
db.SubmitChanges();
}
}

I would suggest letting SQL Server deal with the concurrency.
Here's how:
Create a stored procedure that accepts your log values (method name, month/date, and execution statistics) as arguments.
In the stored procedure, before anything else, get an application lock as described here, and here. Now you can be sure only one instance of the stored procedure will be running at once. (Disclaimer! I have not tried sp_getapplock myself. Just saying. But it seems fairly straightforward, given all the examples out there on the interwebs.)
Next, in the stored procedure, query the log table for a current-month's entry for the method to determine whether to insert or update, and then do the insert or update.
As you may know, in VS you can drag stored procedures from the Server Explorer into the DBML designer for easy access with LINQ to SQL.
If you're trying to avoid stored procedures then this solution obviously won't be for you, but it's how I'd solve it easily and quickly. Hope it helps!

If you don't want to use the stored procedure approach, a crude way of dealing with it would simply be retrying on that specific exception. E.g:
int maxRetryCount = 5;
for (int i = 0; i < maxRetryCount; i++)
{
try
{
QueryAndUpdateDB();
break;
}
catch(RowUpdateException ex)
{
if (i == maxRetryCount) throw;
}
}

I have not used the sp_getapplock, instead I have used HOLDLOCK and ROWLOCK as seen below:
CREATE PROCEDURE [dbo].[UpdateStatistics]
#MethodName as varchar(50) = null,
#CountryID as varchar(2) = null,
#TokenType as varchar(5) = null,
#Year as int,
#Month as int,
#Count bigint,
#TotalTimeValue bigint
AS
BEGIN
SET NOCOUNT ON;
BEGIN TRAN
UPDATE dbo.[Statistics]
WITH (HOLDLOCK, ROWLOCK)
SET Count = Count + #Count
WHERE MethodName=#MethodName and CountryID=#CountryID and TokenType=#TokenType and Year=#Year and Month=#Month
IF ##ROWCOUNT=0
INSERT INTO dbo.[Statistics] (MethodName, CountryID, TokenType, TotalTimeValue, Year, Month, Count) values (#MethodName, #CountryID, #TokenType, #TotalTimeValue, #Year, #Month, #Count)
COMMIT TRAN
END
GO
I have tested it by calling my web service methods by multiple threads simultaneous and each call is logged without any problems.

Related

Concurrent Read/Write MySQL EF Core

Using EF Core 2.2.6 and Pomelo.EntityFrameworkCore.MySql 2.2.6 (with MySqlConnector 0.59.2)). I have a model for UserData:
public class UserData
{
[DatabaseGenerated(DatabaseGeneratedOption.None)]
public ulong ID { get; private set; }
[Required]
public Dictionary<string, InventoryItem> Inventory { get; set; }
public UserData()
{
Data = new Dictionary<string, string>();
}
}
I have a REST method that can be called that will add items to the user inventory:
using (var transaction = context.Database.BeginTransaction())
{
UserData data = await context.UserData.FindAsync(userId);
// there is code here to detect duplicate entries/etc, but I've removed it for brevity
foreach (var item in items) data.Inventory.Add(item.ItemId, item);
context.UserData.Update(data);
await context.SaveChangesAsync();
transaction.Commit();
}
If two or more calls to this method are made with the same user id then I get concurrent accesses (despite the transaction). This causes the data to sometimes be incorrect. For example, if the inventory is empty and then two calls are made to add items simultaneously (item A and item B), sometimes the database will only contain either A or B, and not both. From logging it appears that it is possible for EF to read from the database while the other read/write is still occurring, causing the code to have the incorrect state of the inventory for when it tries to write back to the db. So I tried marking the isolation level as serializable.
using (var transaction = context.Database.BeginTransaction(System.Data.IsolationLevel.Serializable))
Now I sometimes see an exception:
MySql.Data.MySqlClient.MySqlException (0x80004005): Deadlock found when trying to get lock; try restarting transaction
I don't understand how this code could deadlock... Anyways, I tried to proceed by wrapping this whole thing in a try/catch, and retry:
public static async Task<ResponseError> AddUserItem(Controller controller, MyContext context, ulong userId, List<InventoryItem> items, int retry = 5)
{
ResponseError result = null;
try
{
using (var transaction = context.Database.BeginTransaction(System.Data.IsolationLevel.Serializable))
{
UserData data = await context.UserData.FindAsync(userId);
// there is code here to detect duplicate entries/etc, but I've removed it for brevity
foreach (var item in items) data.Inventory.Add(item.ItemId, item);
context.UserData.Update(data);
await context.SaveChangesAsync();
transaction.Commit();
}
}
catch (Exception e)
{
if (retry > 0)
{
await Task.Delay(SafeRandomGenerator(10, 500));
return await AddUserItem(controller, context, userId, items, retry--);
}
else
{
// store exception and return error
}
}
return result;
}
And now I am back to the data being sometimes correct, sometimes not. So I think the deadlock is another problem, but this is the only method accessing this data. So, I'm at a loss. Is there a simple way to read from the database (locking the row in the process) and then writing back (releasing the lock on write) using EF Core? I've looked at using concurrency tokens, but this seems overkill for what appears (on the surface to me) to be a trivial task.
I added logging for mysql connector as well as asp.net server and can see the following failure:
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
=> RequestId:0HLUD39EILP3R:00000001 RequestPath:/client/AddUserItem => Server.Controllers.ClientController.AddUserItem (ServerSoftware)
Failed executing DbCommand (78ms) [Parameters=[#p1='?' (DbType = UInt64), #p0='?' (Size = 4000)], CommandType='Text', CommandTimeout='30']
UPDATE `UserData` SET `Inventory` = #p0
WHERE `ID` = #p1;
SELECT ROW_COUNT();
A total hack is to just delay the arrival of the queries by a bit. This works because the client is most likely to generate these calls on load. Normally back-to-back calls aren't expected, so spreading them out in time by delaying on arrival works. However, I'd rather find a correct approach, since this just makes it less likely to be an issue:
ResponseError result = null;
await Task.Delay(SafeRandomGenerator(100, 500));
using (var transaction = context.Database.BeginTransaction(System.Data.IsolationLevel.Serializable))
// etc
This isn't a good answer, because it isn't what I wanted to do, but I'll post it here as it did solve my problem. My problem was that I was trying to read the database row, modify it in asp.net, and then write it back, all within a single transaction and while avoiding deadlocks. The backing field is JSON type, and MySQL provides some JSON functions to help modify that JSON directly in the database. This required me to write SQL statements directly instead of using EF, but it did work.
The first trick was to ensure I could create the row if it didn't exist, without requiring a transaction and lock.
INSERT INTO UserData VALUES ({0},'{{}}','{{}}') ON DUPLICATE KEY UPDATE ID = {0};
I used JSON_REMOVE to delete keys from the JSON field:
UPDATE UserData as S set S.Inventory = JSON_REMOVE(S.Inventory,{1}) WHERE S.ID = {0};
and JSON_SET to add/modify entries:
UPDATE UserData as S set S.Inventory = JSON_SET(S.Inventory,{1},CAST({2} as JSON)) WHERE S.ID = {0};
Note, if you're using EF Core and want to call this using FromSql then you need to return the entity as part of your SQL statement. So you'll need to add something like this to each SQL statement:
SELECT * from UserData where ID = {0} LIMIT 1;
Here is a full working example as an extension method:
public static async Task<UserData> FindOrCreateAsync(this IQueryable<UserData> table, ulong userId)
{
string sql = "INSERT INTO UserData VALUES ({0},'{{}}','{{}}') ON DUPLICATE KEY UPDATE ID = {0}; SELECT * FROM UserData WHERE ID={0} LIMIT 1;";
return await table.FromSql(sql, userId).SingleOrDefaultAsync();
}
public static async Task<UserData> JsonRemoveInventory(this DbSet<UserData> table, ulong userId, string key)
{
if (!key.StartsWith("$.")) key = $"$.\"{key}\"";
string sql = "UPDATE UserData as S set S.Inventory = JSON_REMOVE(S.Inventory,{1}) WHERE S.ID = {0}; SELECT * from UserData where ID = {0} LIMIT 1;";
return await table.AsNoTracking().FromSql(sql, userId, key).SingleOrDefaultAsync();
}
Usage:
var data = await context.UserData.FindOrCreateAsync(userId);
await context.UserData.JsonRemoveInventory(userId, itemId);

how NamedParameterJdbcTemplate.update really works with Spring and MySQL

Ok, I've probably dug up the entire Google land and still couldn't find anything that could possibly answer my question.
I have my little foo method that does some deleting like this:
private void foo()
{
jdbcNamedParameterTemplate.update(sqlString, params); //1
jdbcNamedParameterTemplate.update(sqlString2, params2); //2
}
sqlString and sqlString2 are just delete statements like "Delete * from FooBar".
So when I get to the second call to update, do I have any guarantee that whatever operation the first one invokes in the database has already finished?
If you do that two in one session, and non multithreading, then yes the first one invokes in the database has already finished before the second update.
But if not in the same session you can check the version to check if the object already changed or not
int oldVersion = foo.getVersion();
session.load( foo, foo.getKey() ); // load the current state
if ( oldVersion != foo.getVersion()) { .... }// if true then the object has been changed

Enable hbm2ddl.keywords=auto-quote in Fluent NHibernate

I have made a tiny software tool that allows me to display or run SQL generated from NHibernate. I made this because hbm2ddl.auto is not recommended for production.
I have one problem: when I generate the SQL I always get the infamous Index column unquoted, because I need .AsList() mappings. This prevents me to run the SQL.
In theory, if I had an XML configuration of NHibernate I could use hbm2ddl.keywords tag, but unfortunately since my tool is designed as a DBA-supporting tool for multiple environments, I must use a programmatic approach.
My approach (redundant) is the following:
private static Configuration BuildNHConfig(string connectionString, DbType dbType, out Dialect requiredDialect)
{
IPersistenceConfigurer persistenceConfigurer;
switch (dbType)
{
case DbType.MySQL:
{
persistenceConfigurer =
MySQLConfiguration
.Standard
.Dialect<MySQL5Dialect>()
.Driver<MySqlDataDriver>()
.FormatSql()
.ShowSql()
.ConnectionString(connectionString);
requiredDialect = new MySQL5Dialect();
break;
}
case DbType.MsSqlAzure:
{
persistenceConfigurer = MsSqlConfiguration.MsSql2008
.Dialect<MsSqlAzure2008Dialect>()
.Driver<SqlClientDriver>()
.FormatSql()
.ShowSql()
.ConnectionString(connectionString);
requiredDialect = new MsSqlAzure2008Dialect();
break;
}
default:
{
throw new NotImplementedException();
}
}
FluentConfiguration fc = Fluently.Configure()
.Database(persistenceConfigurer)
.ExposeConfiguration(
cfg => cfg.SetProperty("hbm2ddl.keywords", "keywords")
.SetProperty("hbm2ddl.auto", "none"))
.Mappings(
m => m.FluentMappings.AddFromAssemblyOf<NHibernateFactory>());
Configuration ret = fc.BuildConfiguration();
SchemaMetadataUpdater.QuoteTableAndColumns(ret);
return ret;
}
...
public static void GenerateSql(MainWindowViewModel viewModel)
{
Dialect requiredDialect;
Configuration cfg = BuildNHConfig(viewModel.ConnectionString, viewModel.DbType.Value, out requiredDialect);
StringBuilder sqlBuilder = new StringBuilder();
foreach (string sqlLine in cfg.GenerateSchemaCreationScript(requiredDialect))
sqlBuilder.AppendLine(sqlLine);
viewModel.Sql = sqlBuilder.ToString();
}
Explanation: when I want to set the ViewModel's SQL to display on a TextBox (yea, this is WPF) I initialize the configuration programmatically with connection string given in ViewModel and choose the dialect/provider accordingly. When I Fluently Configure NHibernate I both set hbm2ddl.keywords (tried both auto-quote and keywords, this being the default) and, following this blog post, I also use the SchemaMetadataUpdater.
The result is that I'm always presented with SQL like
create table `OrderHistoryEvent` (Id BIGINT NOT NULL AUTO_INCREMENT, EventType VARCHAR(255) not null, EventTime DATETIME not null, EntityType VARCHAR(255), Comments VARCHAR(255), Order_id VARCHAR(255), Index INTEGER, primary key (Id))
where the guilty Index column is not quoted.
The question is: given a programmatic and fluent configuration of NHibernate, how do I tell NHibernate to quote any reserved word in the SQL exported by GenerateSchemaCreationScript?
I have found a workaround: when I generate the update script (the one that runs with hbm2ddl.auto=update) the script is correctly quoted.
The infamous Index column has been already discussed and from my findings it's hardcoded in FNH (ToManyBase.cs, method public T AsList()).
Since the update script is a perfectly working creational script on an empty database, changing the code to generate an update script on an empty DB should equal generating a creational script.
This happens only because I want to generate the script on my own. There is probably a bug in NHibernate that only activates when you call GenerateSchemaCreationScript and not when you let your SessionFactory build the DB for you

Dapper And System.Data.OleDb DbType.Date throwing 'OleDbException : Data type mismatch in criteria expression'

Not sure if I should raise an issue regarding this, so thought I would ask if anybody knew a simple workaround for this first. I am getting an error when I try to use Dapper with OleDbConnection when used in combination with MS Access 2003 (Jet.4.0) (not my choice of database!)
When running the test code below I get an exception 'OleDbException : Data type mismatch in criteria expression'
var count = 0;
using (var conn = new OleDbConnection(connString)) {
conn.Open();
var qry = conn.Query<TestTable>("select * from testtable where CreatedOn <= #CreatedOn;", new { CreatedOn = DateTime.Now });
count = qry.Count();
}
I believe from experience in the past with OleDb dates, is that when setting the DbType to Date, it then changes internally the value for OleDbType property to OleDbTimeStamp instead of OleDbType.Date. I understand this is not because of Dapper, but what 'could' be considered a strange way of linking internally in the OleDbParameter class
When dealing with this either using other ORMs, raw ADO or my own factory objects, I would clean up the command object just prior to running the command and change the OleDbType to Date.
This is not possible with Dapper as far as I can see as the command object appears to be internal. Unfortunately I have not had time to learn the dynamic generation stuff, so I could be missing something simple or I might suggest a fix and contribute rather than simply raise an issue.
Any thoughts?
Lee
It's an old thread but I had the same problem: Access doesn't like DateTime with milliseconds, so you have to add and extension method like this :
public static DateTime Floor(this DateTime date, TimeSpan span)
{
long ticks = date.Ticks / span.Ticks;
return new DateTime(ticks * span.Ticks, date.Kind);
}
And use it when passing parameters:
var qry = conn.Query<TestTable>("select * from testtable where CreatedOn <= #CreatedOn;", new { CreatedOn = DateTime.Now.Floor(TimeSpan.FromSeconds(1)) });
Unfortunately, with current Dapper version (1.42), we cannot add custom TypeHandler for base types (see #206).
If you can modify Dapper (use the cs file and not the DLL) merge this pull request and then you do not have to use Floor on each parameters :
public class DateTimeTypeHandler : SqlMapper.TypeHandler<DateTime>
{
public override DateTime Parse(object value)
{
if (value == null || value is DBNull)
{
return default(DateTime);
}
return (DateTime)value;
}
public override void SetValue(IDbDataParameter parameter, DateTime value)
{
parameter.DbType = DbType.DateTime;
parameter.Value = value.Floor(TimeSpan.FromSeconds(1));
}
}
SqlMapper.AddTypeHandler<DateTime>(new DateTimeTypeHandler());

Generate non-identity primary key

My workplace doesn't use identity columns or GUIDs for primary keys. Instead, we retrieve "next IDs" from a table as needed, and increment the value for each insert.
Unfortunatly for me, LINQ-TO-SQL appears to be optimized around using identity columns. So I need to query and update the "NextId" table whenever I perform an insert. For simplicity, I do this during the creation of the new object:
var db = new DataContext( "...connection string..." );
var car = Car
{
Id = GetNextId<Car>( db ),
TopSpeed = 88.0
};
db.InsertOnSubmit( car );
db.SubmitChanges();
The GetNextId method is something like this:
public int GetNextId<T>( DataContext db )
{
using ( var transaction = new TransactionScope ( TransactionScopeOption.RequiresNew ) )
{
var nextId = (from n in db.GetTable<NextId> ()
where n.TableName == typeof(T).Name
select n).Single ();
nextId.Value += 1;
db.SubmitChanges ();
transaction.Complete ();
return nextId.Value - 1;
}
}
Since all operations between creation of the data context and the call to SubmitChanges are part of one transaction, do I need to create a separate data context for retrieving next IDs? Each time I need an ID, I need to query and update a table inside a transaction to prevent multiple apps from grabbing the same value. The call to SubmitChanges() in the GetNextId() method would submit all previous operations, which I don't want to do.
Is a separate data context the only way, or is there something better I could try?