Memory leak when inserting into MySQL with Powershell v4 - mysql

Question

I'm using powershell v4 on W2K12 R2 (fully patched) to insert a large number(100+ million) of records into a MySQL database. I've run into a bit of a problem where memory usage continues growing and growing despite aggressively removing variables and garbage collecting. Note that the memory usage is growing on the box that I'm running the script on -not the DB server.
The insertion speed is good and the job runs fine. However, I have a memory leak and have been beating my head against a wall for a week trying to figure out why. I know from testing that the memory accumulates when calling the MySQL portion of the script and not anywhere else.
I've noticed that after every insertion that the memory grows from anywhere between 1MB and 15MB.
Here is the basic flow of the process (code at the bottom).
-records are being added to an array until there are 1,000 records in the array
-once there are a thousand records, they are inserted, as a batch, into the DB
-the array is then emptied using the .clear() method (I've verified that 0 records remain in array).
-I've tried aggressively garbage collecting after every insert (no luck there).
-also tried removing variables and then garbage collecting. Still no luck.
The code below is simplified for the sake of brevity. But, it shows how I'm iterating over the records and doing the insert:
$reader = [IO.File]::OpenText($filetoread)
$lineCount = 1
while ($reader.Peek() -ge 0) {
if($lineCount -ge 1000-or $reader.Peek() -lt 0) {
insert_into_db
$lineCount = 0
}
$lineCount++
}
$reader.Close()
$reader.Dispose()
One call to establish the connection:
[void][system.reflection.Assembly]::LoadFrom("C:\Program Files (x86)\MySQL\MySQL Connector Net 6.8.3\Assemblies\v4.5\MySql.Data.dll")
$connection = New-Object MySql.Data.MySqlClient.MySqlConnection($connectionString)
And here is the call to MySQL to do the actual inserts for each 1,000 records:
function insert_into_db {
$command = $connection.CreateCommand() # Create command object
$command.CommandText = $query # Load query into object
$script:RowsInserted = $command.ExecuteNonQuery() # Execute command
$command.Dispose() # Dispose of command object
$command = $null
$query = $null
}
If anyone has any ideas or suggestions I'm all ears!
Thanks,
Jeremy

Answer 1

My initial conclusion about the problem being related to the Powershell -join operator appear to be wrong.
Here is what I was doing. Note that I'm adding each line to an array, which I will un-roll later when I form my SQL. (On a side note, adding items to an array tends to more performant than concatenating strings)
$dataForInsertion = = New-Object System.Collections.Generic.List[String]
$reader = [IO.File]::OpenText($filetoread)
$lineCount = 1
while ($reader.Peek() -ge 0) {
$line = $reader.Readline()
$dataForInsertion.add($line)
if($lineCount -ge 1000-or $reader.Peek() -lt 0) {
insert_into_db -insertthis $dataForInsertion
$lineCount = 0
}
$lineCount++
}
$reader.Close()
$reader.Dispose()
Calling the insert function:
sql_query -query "SET autocommit=0;INSERT INTO ``$table`` ($columns) VALUES $($dataForInsertion -join ',');COMMIT;"
The improved insert function now looks like this:
function insert_into_db {
$command.CommandText = $query # Load query into object
$script:RowsInserted = $command.ExecuteNonQuery() # Execute command
$command.Dispose() # Dispose of command object
$query = $null
}
So, it turns out my initial conclusion about the source of the problem was wrong. the Powershell -join operator had nothing to do with the issue.
In my SQL insert function I was repeatedly calling $connection.CreateCommand() on every insert. Once I moved that into the function that handles setting up the connection (which is only called once -or when needed) the memory leak disappeared.