Laravel - Chunk update too slow (SQLSRV - MYSQL) - mysql

I have a function in controller that selects the entire table Clients of a SQL Server database (remote, with less than 1 ms connection), and updates the Clients table in my MySQL DB, there are approx. 15000 rows.
When it was on development in my computer (Windows 10), used to take 24-26 seconds to complete the update. Now in the production server, it takes between 8 and 10 minutes. I tested the connection between my server and the SQL Server and is really fast, so the problem is not the SQL driver but MySQL.
Server: Ubuntu 18.04 LTS, CPU usage2.13% of 4 CPU(s), Memory usage 79.13% (6.26 GiB of 7.91 GiB)
MySQL Ver 5.7.32
Mytop: 50 qps (now) when running the fuction, Slow qps: 0.0, Key efficiency 50% (it was 97% before restarting MYSQL service)
Code
ClienteAsesorSql::where('Concepto', 'NOT LIKE', '%RESERVADO%')
->chunk(500, function ($clientes_asesor) {
foreach ($clientes_asesor as $clientes) {
$cliente_opyme = Cliente::updateOrCreate(['cli_id' => $clientes->ClienteId],[
'cli_id' => $clientes->ClienteId,
'cli_nombre' => $clientes->Concepto,
'cli_email' => $clientes['e-mail'],
'cli_telefono' => $clientes->Telefono,
'cli_movil' => $clientes->SMS,
'cli_domicilio' => $clientes->DomicilioNoFiscal,
'created_at' => $clientes->FechaEstado
]);
$cliente_opyme ->save();
}
});

Related

Terraform 11 AWS RDS Upgrade Fails

I am using Terraform v11 to upgrade DB's running on AWS RDS with MySql engine from v5.6 to v5.7.
Each of the DB's has it's own dedicated options group and parameter group.
The upgrade process fails with error:
aws_db_parameter_group.manager: Error deleting DB parameter group: InvalidDBParameterGroupState: One or more database instances are still members of this parameter group foo-rds-mysql56, so the group cannot be deleted
Is there any workaround for this?
Reply to ydaetskcoR:
The plan output is :
-/+ aws_db_parameter_group.foo (new resource required)
id: "foo-mysql-56" => <computed> (forces new resource)
arn: "arn:aws:rds:eu-west-2:238425939713:pg:foo-mysql-56" => <computed>
description: "Managed by Terraform" => "Managed by Terraform"
family: "mysql5.6" => "mysql5.7" (forces new resource)
name: "foo-mysql-56" => "foo-mysql-57" (forces new resource)
name_prefix: "" => <computed>
parameter.#: "1" => "1"
parameter.2547865204.apply_method: "pending-reboot" => "pending-reboot"
parameter.2547865204.name: "max_connections" => "max_connections"
parameter.2547865204.value: "128" => "128"

Use of undefined constant SIGKILL - assumed 'SIGKILL' in Laravel 5.7 queue

I'm getting this error "Use of undefined constant SIGKILL - assumed 'SIGKILL'" from my AJAX request, that starts this artisan command ->
Artisan::call('queue:work', [
'connection' => 'database',
'--memory' => '700',
'--tries' => '1',
'--timeout' => '35000',
'--queue' => 'updates'
]);
I'm using Laravel 5.7 as framework for application.
Jobs are managed from database, configuration ->
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 18000,
],
Problem appeared recently.. That is weird, because troubles wasn't here before and all this "system" worked just fine. Now worker get some jobs done just fine, but then it drops to error, and writes to table "failed_jobs" in DB this ->
ErrorException: PDOStatement::execute(): MySQL server has gone away in /srv/migration-xxxxxx-xxxx-xxxxxx/www/vendor/laravel/framework/src/Illuminate/Database/Connection.php:458
As DB I'm using Microsoft Azure MySQL DB. Microsoft specialist after consultation find nothing .. server is working correctly. Queries are just fine, not that big to fail.
Please help, don't know what to do, or what is wrong...

PHP 7 Performance

I've tried to reproduce this benchmark which compares PHP 7 with older versions on a Wordpress server: http://talks.php.net/oz15#/wpbench
My configuration is nearly the same, the server has an i7, SSD, 16GB RAM and debian. The server software is nginx. Suprisingly my results differ a lot from the ones linked above.
In my tests Siege (https://www.joedog.org/siege-home/) outputs the following:
For PHP 7.0.0RC1:
siege -c100 -r100 http://10.22.255.133/wordpress/
** SIEGE 3.0.8
** Preparing 100 concurrent users for battle.
The server is now under siege.. done.
Transactions: 10000 hits
Availability: 100.00 %
Elapsed time: 131.61 secs
Data transferred: 95.77 MB
Response time: 0.75 secs
Transaction rate: 75.98 trans/sec
Throughput: 0.73 MB/sec
Concurrency: 56.98
Successful transactions: 10000
Failed transactions: 0
Longest transaction: 1.01
Shortest transaction: 0.04
For PHP 5.6.12:
siege -c100 -r100 http://10.22.255.133/wordpress/
** SIEGE 3.0.8
** Preparing 100 concurrent users for battle.
The server is now under siege.. done.
Transactions: 10000 hits
Availability: 100.00 %
Elapsed time: 63.41 secs
Data transferred: 95.77 MB
Response time: 0.03 secs
Transaction rate: 157.70 trans/sec
Throughput: 1.51 MB/sec
Concurrency: 4.45
Successful transactions: 10000
Failed transactions: 0
Longest transaction: 0.63
Shortest transaction: 0.01
When looking at the transaction rate you can see, that PHP 5 is about two times faster than PHP 7. I can't believe that.
Another interesting fact is, that running this benchmark (http://www.php-benchmark-script.com/) results in PHP 7 being about 3 times faster than PHP 5 (of course on the same server where I've also tested Wordpress). The measured results were:
PHP 7.0.0RC1 | PHP 5.5.28
Math: 0.201 | 0.683
String Manipulation: 0.271 | 0.77
Loops: 0.166 | 0.486
If Else: 0.12 | 0.295
I've uploaded both phpinfo() files in case that helps:
PHP Version 7.0.0RC1: http://simsso.de/downloads/stackoverflow/php7.html
PHP Version 5.6.12-0+deb8u1: http://simsso.de/downloads/stackoverflow/php5.html
Do you have any idea why PHP 7 is that much slower in my tests with Wordpress?
With opcache enabled PHP 7 is actually twice as fast as PHP 5. Thanks Mjh for your hint!
I've made the following measurements on a randomly filled WordPress Server.
Siege now outputs the following for PHP 7.0.0RC1:
Transactions: 10000 hits
Availability: 100.00 %
Elapsed time: 62.14 secs
Data transferred: 604.20 MB
Response time: 0.02 secs
Transaction rate: 160.93 trans/sec
Throughput: 9.72 MB/sec
Concurrency: 3.77
Successful transactions: 10000
Failed transactions: 0
Longest transaction: 0.41
Shortest transaction: 0.01
And PHP 5.6.12:
siege -c100 -r100 http://10.22.255.133/wordpress/
** SIEGE 3.0.8
** Preparing 100 concurrent users for battle.
The server is now under siege.. done.
Transactions: 10000 hits
Availability: 100.00 %
Elapsed time: 119.98 secs
Data transferred: 604.20 MB
Response time: 0.60 secs
Transaction rate: 83.35 trans/sec
Throughput: 5.04 MB/sec
Concurrency: 49.86
Successful transactions: 10000
Failed transactions: 0
Longest transaction: 4.06
Shortest transaction: 0.04
According to the output of phpinfo you posted, opcache isn't enabled for your PHP 7, while it is for PHP 5. That alone can amount for a huge difference.
I currently have the same surprising results on the CLI side.
One of my old projects uses a PHING build. It was running on PHP 5.3 then PHP 5.6.
I tried using PHP 7 and noticed a huge difference. So i decided to time the script execution.
FYI it is a real life projects with thousands of files processed during the build.
Build using PHP 5.3.29:
3 minutes and 44 seconds elapsed.
Build using PHP 7.2.11:
11 minutes and 41 seconds elapsed.
I noticed the CLI did not have opcache activated, here is the results with opcache:
Build using PHP 7.2.11 + opcache:
12 minutes and 18 seconds elapsed.
Yes, WORSE
FYI:
$ php --info |grep opcache
opcache.blacklist_filename => no value => no value
opcache.consistency_checks => 0 => 0
opcache.dups_fix => Off => Off
opcache.enable => On => On
opcache.enable_cli => On => On
opcache.enable_file_override => Off => Off
opcache.error_log => no value => no value
opcache.file_cache => no value => no value
opcache.file_cache_consistency_checks => 1 => 1
opcache.file_cache_only => 0 => 0
opcache.file_update_protection => 2 => 2
opcache.force_restart_timeout => 180 => 180
opcache.huge_code_pages => Off => Off
opcache.inherited_hack => On => On
opcache.interned_strings_buffer => 8 => 8
opcache.lockfile_path => /tmp => /tmp
opcache.log_verbosity_level => 1 => 1
opcache.max_accelerated_files => 10000 => 10000
opcache.max_file_size => 0 => 0
opcache.max_wasted_percentage => 5 => 5
opcache.memory_consumption => 128 => 128
opcache.opt_debug_level => 0 => 0
opcache.optimization_level => 0x7FFFBFFF => 0x7FFFBFFF
opcache.preferred_memory_model => no value => no value
opcache.protect_memory => 0 => 0
opcache.restrict_api => no value => no value
opcache.revalidate_freq => 2 => 2
opcache.revalidate_path => Off => Off
opcache.save_comments => 1 => 1
opcache.use_cwd => On => On
opcache.validate_permission => Off => Off
opcache.validate_root => Off => Off
opcache.validate_timestamps => On => On
Btw, I have to say I never noticed a huge difference on prod with apache when switched from PHP 5 to PHP 7. Despites all of the bechmarks we see online, the difference is far from obvious.
Neddless to say, for that project, I will stick to PHP 5 version.

Silverstripe losing connection to MySQL server requiring a restart

Firstly please apologies for how vague this post may be but I'm completely at a loss as to what could be occurring here.
I've been having an issue with a site I've developed for a client where it seemingly, randomly loses connection to the MySQL server and requires a reboot of the server for the site to become available again.. I've a number of other SS sites in my portfolio and not a single other one has experienced this before.
The site can run for a few days or an hour or sometimes a couple of weeks before having this issue and I've not been able to pin down what actually triggers it, though it appears to be related to some kind of usage of the CMS. The client will report that it will lose the connection when he's trying to upload a photo or write a blog post. I've posted the error at the bottom of the post to not clutter things.
Thinking it was an issue with the server set up, I set up a brand new digital ocean droplet and migrated the site over to there, but the issue remains so it leads me to believe it's something within the framework or my site config (old server was running on CentOs and the new one is Ubuntu).
This is the only site I have set up using composer. A novice to composer, I followed the instructions on the SS site to achieve this. My composer.json is below. Does it look like I have any issues here?:
{
"name": "silverstripe/installer",
"description": "The SilverStripe Framework Installer",
"require": {
"php": ">=5.3.2",
"silverstripe/cms": "3.1.6",
"silverstripe/framework": "3.1.6",
"silverstripe-themes/simple": "*",
"silverstripe/blog": "dev-master",
"undefinedoffset/sortablegridfield": "dev-master"
},
"config": {
"process-timeout": 600
},
"minimum-stability": "dev"
}
I've not made any changes to the core. Nothing unusual in there but my _config.php is:
<?php
global $project;
$project = 'mysite';
global $databaseConfig;
$databaseConfig = array(
"type" => 'MySQLDatabase',
"server" => 'localhost',
"username" => 'root',
"password" => 'xxxxxx',
"database" => 'mysite',
"path" => '',
);
// Set the site locale
i18n::set_locale('en_US');
Director::set_environment_type('dev');
Security::setDefaultAdmin("fraser","xxxxxxx");
Can anyone think of any reason why the connection would be dropping to the MySQL server? Is there anything else I should check? Do you need any more information from me?
[Warning] mysqli::mysqli(): (HY000/2003): Can't connect to MySQL server on '127.0.0.1' (111)
GET /
Line 68 in /var/www/mysite/framework/model/MySQLDatabase.php
Source
59 * - password: The password to log on with
60 * - database: The database to connect to
61 * - timezone: (optional) The timezone offset. For example: +12:00, "Pacific/Auckland", or "SYSTEM"
62 */
63 public function __construct($parameters) {
64 if(!empty($parameters['port'])) {
65 $this->dbConn = new MySQLi($parameters['server'], $parameters['username'], $parameters['password'],
66 '', $parameters['port']);
67 } else {
68 $this->dbConn = new MySQLi($parameters['server'], $parameters['username'], $parameters['password']);
69 }
70
71 if($this->dbConn->connect_error) {
72 $this->databaseError("Couldn't connect to MySQL database | " . $this->dbConn->connect_error);
73 }
74
Trace
mysqli->mysqli(<filtered>,<filtered>,<filtered>)
MySQLDatabase.php:68
MySQLDatabase->__construct(Array)
DB.php:174
DB::connect(<filtered>)
main.php:127
{closure}(ErrorControlChain)
call_user_func(Closure,ErrorControlChain)
ErrorControlChain.php:125
ErrorControlChain->step()
ErrorControlChain.php:117
ErrorControlChain->execute()
main.php:154
require_once(/var/www/mysite/framework/main.php)
index.php:65
[User Error] Couldn't connect to MySQL database | Can't connect to MySQL server on '127.0.0.1' (111)
GET /
Line 598 in /var/www/mysite/framework/model/MySQLDatabase.php
Source
589 }
590
591 public function databaseError($msg, $errorLevel = E_USER_ERROR) {
592 // try to extract and format query
593 if(preg_match('/Couldn\'t run query: ([^\|]*)\|\s*(.*)/', $msg, $matches)) {
594 $formatter = new SQLFormatter();
595 $msg = "Couldn't run query: \n" . $formatter->formatPlain($matches[1]) . "\n\n" . $matches[2];
596 }
597
598 user_error($msg, $errorLevel);
599 }
600
601 /**
602 * Return a boolean type-formatted string
603 *
604 * #param array $values Contains a tokenised list of info about this data type
Trace
Couldn't connect to MySQL database | Can't connect to MySQL server on '127.0.0.1' (111)
MySQLDatabase.php:598
MySQLDatabase->databaseError(Couldn't connect to MySQL database | Can't connect to MySQL server on '127.0.0.1' (111))
MySQLDatabase.php:72
MySQLDatabase->__construct(Array)
DB.php:174
DB::connect(<filtered>)
main.php:127
{closure}(ErrorControlChain)
call_user_func(Closure,ErrorControlChain)
ErrorControlChain.php:125
ErrorControlChain->step()
ErrorControlChain.php:117
ErrorControlChain->execute()
main.php:154
require_once(/var/www/mysite/framework/main.php)
index.php:65

pdo dblib stored procedure insert wrong characters

i'm using microsoft sql server as database and my php code on centos server. using freetds and dblib to connect from yii framework to mssql.
everything's are fine. after insert into db by Stored Procedure, data saved but in database query we have ??????? in this NVARCHAR column.
my data in utf-8 arabic. this is my configuration and code:
fretds.conf
[mss]
host = 172.31.1.2
ip = 172.31.1.2
port = 1433
tds version = 7.0
yii main configuration
'db'=>array(
'connectionString' => 'dblib:host=mss;database=XXXX;charset=utf8',
'username' => 'XXX',
'password' => 'XXXXXXXX',
'charset' => 'utf8',
'tablePrefix' => 'tbl_',
'enableProfiling' => true,
'schemaCachingDuration' => 5 * 60 * 60,
),
my model Stored Procedure
$builder=$this->getCommandBuilder();
$table=$this->getMetaData()->tableSchema;
$command=$builder->createSqlCommand('EXEC dbo.sp_link_comment_insert :link_id, :cmnt_parent_id, :user_id, :cmnt_status, :cmnt_text, :cmnt_thread',
array(
':link_id'=>58829,
':cmnt_parent_id'=>'',
':user_id'=>9,
':cmnt_status'=>1,
':cmnt_text'=>'تست ارسال comment',
':cmnt_thread'=>0,
)
);
$command->execute();
A late reply. Try specifying the encoding in the freetds configuration using the option client charset = UTF-8. Your config file would look like this after the change:
[mss]
host = 172.31.1.2
ip = 172.31.1.2
port = 1433
tds version = 7.0
client charset = UTF-8
According to the docs chars not recognized are converted to ?.
If FreeTDS runs into a character it can not convert, its behavior
varies according to the severity of the problem. On retrieving data
from the server, FreeTDS substitutes an ASCII '?' in the character's
place, and emits a warning message stating that some characters could
not be converted.
Reference: http://freetds.schemamania.org/userguide/localization.htm
Your driver (freeTDS) does not support UTF-8 for MSSQL. The only driver that natively supports UTF-8 for MSSQL is SQLDRV driver available only for windows platform.
To store data in UTF-8 using any other than SQLDRV driver (SQLDRV is not available for linux at the moment), the application must take care of converting from source charset into UTF-8 and vice versa. Therefore, setting "charset=UTF8" in your connection string has no effect.
In PHP, you should convert data before storing into MSSQL like so:
$dataUTF8 = iconv('windows-1250', 'utf-8', $data);
$sql = "INSERT INTO table (value) VALUES ($dataUTF8)";
and convert them back when reading from MSSQL:
$sql = "SELECT dataUTF8 FROM table;";
$data = iconv('utf-8', 'windows-1250', $dataUTF8);