EF Core: Update thousands of records failed - exception

Question

I am using EF Core to update thousands of records from a table called MyTable. It already has 1.500.000 rows. And here is the simple code I use to update one of its property called MyProp for each record with a different value for each of them:
using (MyDbContext ctx = new MyDbContext())
{
var rows = ctx.Set<MyTable>().Take(1000000).ToList(); // I only take the first 1 million rows
int idx = 0;
foreach (var row in rows)
{
row.MyProp = $"MyNewValue{idx++}";
}
ctx.SaveChanges();
}
If I take one million rows to update them, while calling ctx.SaveChanges(), I get the following exception: Expected to read 4 header bytes but only received 0. (see below for the full stack trace)
I don't know why am I facing such exception, it appears if I take 1000000 rows but not if I take only 750000 for example.
I read it could be related to the timeout of the SQL server: which are listed below, but I really do not know if it is really related to one timeout and how I can find a solution. Should I divide the update into several ones and call several times SaveChanges()?
Thank you for any suggestions.
Here are the timeouts from the MySql server:
connect_timeout 10
delayed_insert_timeout 300
have_statement_timeout YES
innodb_flush_log_at_timeout 1
innodb_lock_wait_timeout 50
innodb_print_lock_wait_timeout_info OFF
innodb_rollback_on_timeout OFF
interactive_timeout 610
lock_wait_timeout 31536000
net_read_timeout 30
net_write_timeout 60
rpl_stop_slave_timeout 31536000
slave_net_timeout 60
thread_pool_idle_timeout 60
wait_timeout 610
Here is the full stacktrace:
From Microsoft.EntityFrameworkCore.Update : ERROR An exception occurred in the database while saving changes for context type 'MyDbContext'.
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details.
---> MySql.Data.MySqlClient.MySqlException (0x80004005): Failed to read the result set.
---> System.IO.EndOfStreamException: Expected to read 4 header bytes but only received 0.
at MySqlConnector.Protocol.Serialization.ProtocolUtility.DoReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func1 getNextSequenceNumber, ArraySegmentHolder1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\Protocol\Serialization\ProtocolUtility.cs:line 462
at MySqlConnector.Protocol.Serialization.StandardPayloadHandler.ReadPayloadAsync(ArraySegmentHolder`1 cache, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\Protocol\Serialization\StandardPayloadHandler.cs:line 37
at MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 665

Answer 1

For such update better to use temporary table which will hold new values. Then normally records should be updated by join to temporary table.
There is extension for EF Core linq2db.EntityFrameworkCore (note that I'm one of the creators). It will add temporary tables support to LINQ queries and make possible to do such update without leaving type-safe LINQ.
// definition of temporary table structure
class IntermediateTable
{
public int Id {get; set;}
public string MyProp {get; set;}
}
var items = ... // some input list
var rowsToUpdate = items.Select(x => new IntermediateTable
{
Id = x.Id,
MyProp = x.MyProp
});
using var ctx = new MyDbContext();
using var db = ctx.CreateLinqToDBConnection();
// here all records are populated to database as fast as possible
using var tempTable = db.CreateTempTable(rowsToUpdate,
options: new BulkCopyOptions { BulkCopyType = BulkCopyType.ProviderSpecific },
tableName: "SomeTempTableName");
// preparing update query
var queryToUpdate =
from x in ctx.Set<MyTable>()
join t in tempTable on x.Id equals t.Id
select new {x, t};
queryToUpdate
.Set(u => u.x.MyProp, u => u.t.MyProp)
.Update();
At the end Temporary Table will be dropped automatically.

Answer 2

For your usecase since you want to set an explicit value you can execute some direct sql that will not return any rows on the server.
Read into this nice chapter and you can actually just translate the whole operation over to a simple sql statement that will run on db only.
The paging ( skip ) part will be moved to the database, no data will be retrieved saving performance and the whole operation will not consume memory on the server