Adjusting for the default time-zone setting on RDS - mysql

We recently switched to an RDS instance and noticed that bunch of our database tasks were getting triggered 4 hours earlier than needed. On investigating further, the problem is caused by the default time-zone setting (UTC) on the RDS instance. Since this setting can not be altered, we would like to fix the issue on the code level globally across all our applications using this database instance. I tried to set the time-zone on the db instance I create to 'US/Eastern' by using
set GLOBAL time_zone = 'US/Eastern'" OR
set time_zone = 'US/Eastern'"
But that generates an error "Database error: Unknown or incorrect time zone: 'US/Eastern'"
What do you think I am doing wrong here? Does anyone has used any other solutions ?

Unfortunately it's not possible to set the default_timezone in the RDS DB ParameterGroups so your attempt was the right direction already.
$ rds-describe-db-parameters default | grep "time_zone"
DBPARAMETER default_time_zone engine-default string static false
To set the global value via SET GLOBAL you need to have the SUPER privilege which is not granted to you as a RDS user.
The only way to set the time_zone is on a per-connection basis
mysql> SET time_zone = timezone;
On my machines I've tried US/Eastern successfully but I got a quite old generation running.
To determine the timezones you have available log into your box
mysql -h yourboxhost.rds.amazonaws.com -u <youruser> -p
and type
mysql> SELECT * FROM mysql.time_zone_name;
You should get a list of installed and valid timezone names you can set on your instance
+----------------------------------------+--------------+
| Name | Time_zone_id |
+----------------------------------------+--------------+
| Africa/Abidjan | 1 |
| Africa/Accra | 2 |
| Africa/Addis_Ababa | 3 |
| Africa/Algiers | 4 |
| Africa/Asmara | 5 |
| Africa/Asmera | 6 |
| Africa/Bamako | 7 |
| Africa/Bangui | 8 |
| Africa/Banjul | 9 |
| Africa/Bissau | 10 |
| Africa/Blantyre | 11 |
| Africa/Brazzaville | 12 |
| Africa/Bujumbura | 13 |
| Africa/Cairo | 14 |
etc...
You have to set the time_zone each time you connect to your database server
For example if you use the php Mysqli extension you can do this
$mysqli = mysqli_init();
mysqli_options($mysqli,MYSQLI_INIT_COMMAND,"SET time_zone = 'Africa/Brazzaville'" );
mysqli_real_connect($mysqli,$host, $user, $pass,$dbName) or die ('Unable to connect');
Otherwise just manually ( in terms of let your database connector do it ) execute the SET time_zone = '<YOUR_DESIRED_TIMEZONE>' Query right after you've connected to your database

The time_zone setting of RDS database instances can now be modified: https://aws.amazon.com/de/premiumsupport/knowledge-center/rds-change-time-zone/

I did the following steps, So that I could change the timezone
login to RDS and Create New Parameter Group.
Edit the newly created Parameter Group
Set timezone Ex:Asia/Calcutta and Save Changes
Modify RDS instance, change DB's Parameter Group to newly created parameter group
Save And Reboot RDS instance

tldr;
Create a "shared" schema that all your users have EXECUTE access to, create a SPROC that modifies the session timezone and modify the init_connect MySQL parameter to call it.
As Ryan Weir pointed out in his excellent answer in a duplicate question this should probably be avoided if possible. If, however, you are like me and want to implement it for the sake of convenience and sanity then I took Ryan's solution and made a few modifications.
If you have multiple users setup in MySQL with varying permissions then simply putting the sproc in the mysql schema might have problems. To solve this I created a new schema called "shared" and gave all my users EXECUTE access to this schema. I then created the following stored procedure.
DROP PROCEDURE IF EXISTS shared.store_time_zone;
CREATE PROCEDURE shared.`store_time_zone`()
IF NOT (POSITION('rdsadmin#' IN CURRENT_USER()) = 1) THEN
SET SESSION time_zone = 'US/Pacific';
END IF;
I prefer to set 'US/Pacific' to handle daylight savings but you should test this to make sure your MySQL instance recognizes it first. Just execute the following query SET SESSION time_zone = 'US/Pacific'; to make sure it works. To look up your timezone execute SELECT * FROM mysql.time_zone_name;
At this point I recommend testing the permissions before you go modifying the paramter group and potential break everything. Simply connect to the DB (preferably with a user that has low level permissions and/or is commonly used) and execute the following queries.
CALL shared.store_time_zone;
select now();
Hopefully you didn't get any errors and the correct time showed up.
Next you will need to modify the init_connect parameter in the DB Parameter Group that your RDS instance is using. You can do this in the RDS web console, through the API or the command line utility. If you use the command line it will look like this:
$ rds-modify-db-parameter-group PARAMGROUP --parameters "name=init_connect, value='CALL shared.store_time_zone', method=immediate"
If you do it through the web console then you just need to change the value of init_connect.
CALL shared.store_time_zone
Go back to your RDS instance in the web console and scroll the details pane down to the DB Parameter Group. It should say something like (applying) or (in-sync). Once it is (in-sync) go test everything out to make sure there are no problems.
If at this point you run into problems and need to roll things back then I recommend setting the init_connect value to something harmless like:
SET SESSION time_zone = '-00:00';
Setting it back to blank is impossible to do from the web console. See this thread for more details on why one can't restore the empty value for the DB parameter

#Thomas Paine's solution works for me except I had to user user() instead of current_user() as inside the context of init_connect current_user() returns the master RDS user. (By master I do not mean rdsadmin which is the real root user but the user created with the DB instance with most privileges.)

Related

PERSIST vs PERSIST_ONLY (MySQL)

I read Syntax for Persisting System Variables in MySQL documentation about PERSIST and PERSIST_ONLY as shown below:
To persist a global system variable to the mysqld-auto.cnf option file in the data directory, precede the variable name by the PERSIST keyword or the ##PERSIST. qualifier:
SET PERSIST max_connections = 1000;
SET ##PERSIST.max_connections = 1000;
To persist a global system variable to the mysqld-auto.cnf file without setting the global variable runtime value, precede the variable name by the PERSIST_ONLY keyword or the ##PERSIST_ONLY. qualifier:
SET PERSIST_ONLY back_log = 100;
SET ##PERSIST_ONLY.back_log = 100;
It seems like PERSIST sets a global variable runtime value but PERSIST_ONLY doesn't set a global variable runtime value but I don't understand what a global variable runtime value is, so I don't really understand the difference between PERSIST and PERSIST_ONLY.
My questions:
What is the global variable runtime value?
What is the difference between PERSIST and PERSIST_ONLY?
what the global variable runtime value is .. what is the difference between PERSIST and PERSIST_ONLY?
There exists a bunch of global option variables which effects the server. They effects the whole server. Changing the variable value changes the server behavior immediately.
Option variables values are loaded from option files during server start. If some option have no according row in the file then hardcoded default value is used.
Option variables are not reloaded when the server works. Changing the setting in the options file does not effect the server until it restarts.
So when you use PERSIST then new value is written into the option file (and it will be applied during the next server start) and is set to current server settings (the server alters its work according this new setting).
When you use PERSIST_ONLY then new value is written into the option file (and it will be applied during the next server start) but it is not set to current server settings (and current server behavior is not changed).
To set a global variable runtime value, PERSIST doesn't need to restart MySQL while PERSIST_ONLY needs to restart MySQL.
For example, a global variable runtime value is "max_connections" which is "151" by default as shown below:
mysql> SELECT ##GLOBAL.max_connections;
+--------------------------+
| ##GLOBAL.max_connections |
+--------------------------+
| 151 |
+--------------------------+
Now, with PERSIST, if setting "500" to "max_connections" as shown below:
mysql> SET PERSIST max_connections = 500;
Then, "max_connections" is now "500" as shown below:
mysql> SELECT ##GLOBAL.max_connections;
+--------------------------+
| ##GLOBAL.max_connections |
+--------------------------+
| 500 |
+--------------------------+
But with PERSIST_ONLY, if setting "500" to "max_connections" as shown below:
mysql> SET PERSIST_ONLY max_connections = 500;
Then, "max_connections" is still "151" as shown below:
mysql> SELECT ##GLOBAL.max_connections;
+--------------------------+
| ##GLOBAL.max_connections |
+--------------------------+
| 151 |
+--------------------------+
Then, stop, start and login MySQL again:
C:\Windows\System32>net stop MySQL80 && net start MySQL80 && mysql -u root -p
Then finally, "max_connections" is now "500" as shown below:
mysql> SELECT ##GLOBAL.max_connections;
+--------------------------+
| ##GLOBAL.max_connections |
+--------------------------+
| 500 |
+--------------------------+
1 row in set (0.00 sec)

How to change Piwik visit urls in MySQL database?

I have this issue my previous post had lot of visits from facebook and they started to add query string to their outbound urls (eg. ?fbclid=IwAR26j_D60TXYnGASbu27ABBdZduNInguL4mp_nK7eqxm6UklZEpWt8jkZM4) so in my stats each visit from Facebook is unique url (some of them have more then one visit so maybe they are shares). I've added redirect for future visitors:
RewriteCond %{QUERY_STRING} fbclid=.+
RewriteRule (.*) /$1?fbclid= [R=302,L]
but I also want to change the urls in Piwik database (only one last post), but I have trouble to find where urls are saved, I've visited the post from incognito mode (because I have cookie set to not track my visits) so I have one url without value.
Piwik have support, but it's for pay subscription for businesses (I have free version), they also have GitHub issues but those are for bugs, so I don't know where to ask this besides StackOverflow.
What I've tried this:
mysql> select * from piwik_log_action where name = 'jcubic.pl/2018/10/pytania-rekrutacyjne-css.html?fbclid='
# this is post after redirect
+----------+----------------------------------------------------------+------------+------+------------+
| idaction | name | hash | type | url_prefix |
+----------+----------------------------------------------------------+------------+------+------------+
| 2246 | jcubic.pl/2018/10/pytania-rekrutacyjne-css.html?fbclid= | 4170874330 | 1 | 2 |
+----------+----------------------------------------------------------+------------+------+------------+
mysql> update piwik_log_link_visit_action
set idaction_url = 2246
where idaction_url in (select idaction
from piwik_log_action
where name like '%pytania-rekrutacyjne-css.html?fbclid=%' and name <> 'jcubic.pl/2018/10/pytania-rekrutacyjne-css.html?fbclid=')
the query was successful adn the adaction_url get updated for all records. But when I've refreshed the Piwki report, I still get pages with fbclid query string with value.
Does anyone have knowledge about Piwik (old version 3.0.0 before it got renamed to matomo) and know how to change the urls in DB?
It seems that my Piwik installation have delete old logs enabled and Piwk is creating Archive based on logs for Report. They are in its own table with data as blob and according to FAQ you should not re-build the archive if you have delete old logs enabled.
But just in case if you don't have this enabled, here are queries I've executed that should update all data and DB and then you can try to re-process Archive from logs as in FAQ.
Here are queries I've executed:
update piwik_log_visit set visit_exit_idaction_url = 2246
where visit_exit_idaction_url in (select idaction
from piwik_log_action
where name like '%pytania-rekrutacyjne-css.html?fbclid=%' and name <> 'jcubic.pl/2018/10/pytania-rekrutacyjne-css.html?fbclid=')
update piwik_log_visit set visit_entry_idaction_url = 2246
where visit_entry_idaction_url in (select idaction
from piwik_log_action
where name like '%pytania-rekrutacyjne-css.html?fbclid=%' and name <> 'jcubic.pl/2018/10/pytania-rekrutacyjne-css.html?fbclid=')
Also in archive.org there are still docs for DB schema

PHPMyAdmin forces to use ut8mb4 as default collation

Ok, so I've spent the morning trying to change the default collation on my XAMPP setup.
Here's the problem: I'm using Format() in a view, to convert a double into a string
CREATE VIEW `test` AS
SELECT
Format(some_data_table.double_number,0) AS string_result
FROM some_data_table;
When I look at the returned column, its showing as utf8mb4_general_ci.
I've tried all manner of settings in my.ini and phpMyAdmin's config.inc.php
to no avail.
As a last resort, I'm prepared to add the collation parameter to view.
I'd be grateful for any tested solution
Ok - i'm going to post my own answer for anyone else who lands here:
(i had seen this somewhere else, but didn't trust it a t the time because there was no explanation).
When the SQL Format() turns a number into a string, it uses the variable character_set_results.
PMA's Variables Tab was showing this as "utf8" but then on a line below, it was saying (session value) = utf8mb4.
So i was aware that PMA was overriding the server default.
My real problem was that I could find no way to change this override - either by using the [mysqld] skip-character-set-client-handshake setting.. or by editing the php.config.inc file.
Today I had a breakthrough.. I established that if I used the same PMA to connect to and older MySQL server, the problem did not occur.
This suggested to be that PMA was forcing utf8mb4 on newer (capable) servers, but not older ones.
I did a text search of phpmyadmin for the string 'mb4' and found the following code in the class: phpMyAdmin/libraries/DatabaseInterface.class.php
// Skip charsets for Drizzle
if (!PMA_DRIZZLE) {
if (PMA_MYSQL_INT_VERSION > 50503) {
$default_charset = 'utf8mb4';
$default_collation = 'utf8mb4_general_ci';
} else {
$default_charset = 'utf8';
$default_collation = 'utf8_general_ci';
}
the PMA_MYSQL_INT_VERSION > 50503 seems to fit with my theory about older mysql versions, so i've backed up the file and edited the class replacing utf8mb4 with utf8 in this function.
phpMyAdmin is now showing what i want in its variables tab, and the Format() function is now returning what i expect.
(I won't give you a tested solution without a failing test case.)
Here's a possible explanation:
mysql> SELECT FORMAT(2e7, 0);
+----------------+
| FORMAT(2e7, 0) |
+----------------+
| 20,000,000 |
+----------------+
But you are working in a "locale" where the "thousands separator" is ., not ,.
The solution has nothing to do with COLLATION. Instead, look at the arguments to FORMAT().
mysql> SELECT FORMAT(2e7, 0, 'de_DE');
+-------------------------+
| FORMAT(2e7, 0, 'de_DE') |
+-------------------------+
| 20.000.000 |
+-------------------------+
I am guessing that MS Access and MySQL are assuming different "Locales", hence stumbling over the thousands separator, and possibly other differences.
References on Locale:
http://dev.mysql.com/doc/refman/5.7/en/locale-support.html
http://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_lc_time_names

my stopwords list is loaded but not working

I have a modified stopwords list file, which basically took out the word 'alone'. I have updated my /etc/my.cnf
ft_stopword_file=/etc/new_stopwords_list.txt
After restarting my mysql server, I did the following to show that mysql is indeed picking up the new variable.
SHOW VARIABLES LIKE '%ft_stop%'
+------------------+-----------------------------+
| Variable_name | Value |
+------------------+-----------------------------+
| ft_stopword_file | /etc/new_stopwords_list.txt |
+------------------+-----------------------------+
Afterward, I did a REPAIR TABLE to update the index. However, when I do a search, the new setting does not seem to take effect. What am I doing wrong?
What mode are you using?. It could be because of the 50% threshold. Did you you check in this direction.
http://dev.mysql.com/doc/refman/5.0/en/fulltext-fine-tuning.html

automating Cucumber test scenarios for MySQL

I've built an important MySQL database, with a lot of view, triggers, functions and procedures.
It's very hard to test, and to not forget anything, so, I've written Cucumber scenarios for all of the features of my DB (Insert, Select, etc., request on functions an procedures etc., and views)
This help us a lot when we test the behavior of all this, and even before writing view and other code, it's very helpful to determinate want we really want to do.
My problem is: after writing Cucumber features, we all test by hand in a MySQL Shell.
I'm new in BDD/TDD and Agile methods, but I've done some search to know how to make some automation, but found nothing very interesting for my case.
Is there somebody who can provide some interesting way to create automation for this?
I don't know Ruby, but by example, is it possible to use RSPec directly with MySQL (with some examples)?
Or in another language, or any solution you can think of!
Thanks in advance!
[EDIT]
If found some interesting things with RSpec and MySQL:
Mysql Support For Cucumber Nagios
mysql_steps.rb
My problem is: I don't have any knoledge with Ruby, RSPec, etc.
I'm working on it with the excellent "Pick Axe" book, and RSPec book from PragProg
But I will be very grateful for a little example of of RSpec steps given the code below:
The MySQL Procedure
DELIMITER $$
CREATE PROCEDURE `prc_liste_motif` (
IN texte TEXT,
IN motif VARCHAR(255),
OUT nb_motif INT(9),
OUT positions TEXT)
BEGIN
DECLARE ER_SYNTAXE CONDITION FOR SQLSTATE '45000';
DECLARE sousChaine TEXT;
DECLARE positionActuelle INT(9) DEFAULT 1;
DECLARE i INT(9) DEFAULT 1;
IF
LENGTH(motif) > LENGTH(texte)
THEN
SIGNAL ER_SYNTAXE
SET MESSAGE_TEXT =
'Bad Request: Le motif est plus long que le texte.',
MYSQL_ERRNO = 400;
END IF;
SET positions = '';
SET nb_motif = 0;
REPEAT
SET sousChaine = SUBSTRING_INDEX(texte, motif, i);
SET positionActuelle = LENGTH(sousChaine) + 1;
IF
positionActuelle < LENGTH(texte) + 1
THEN
IF
LENGTH(positions) > 0
THEN
SET positions = CONCAT(positions, ',');
END IF;
SET positions = CONCAT(positions, positionActuelle);
SET nb_motif = nb_motif + 1;
END IF;
SET i = i + 1;
UNTIL LENGTH(sousChaine) >= LENGTH(texte)
END REPEAT;
END$$
The Cucumber feature:
Feature: Procedure prc_liste_motif
In order to precess a string according to a given unit
I want to know the number of units present in the chain and their positions
Knowing that the index starts at 1
Background: the database mydatabase in our SGBDR server
Given I have a MySQL server on 192.168.0.200
And I use the username root
And I use the password xfe356
And I use the database mydatabase
Scenario Outline: Using the procedure with good values in parameters
Given I have a procedure prc_liste_motif
And I have entered <texte> for the first parameter
And I have entered <motif> for the second parameter
And I have entered <nb_motif> for the third parameter
And I have entered <positions> for the fourth parameter
When I call prc_liste_motif
Then I should have <out_nb_motif> instead of <nb_motif>
Then I should have <out_positions> instead of <positions>
Exemples:
| texte | motif | nb_motif | positions | out_nb_motif | out_positions |
| Le beau chien | e | | | 3 | 2,5,12 |
| Allo | ll | | | 1 | 2 |
| Allo | w | | | 0 | |
An exemple of passed test by hand in MySQL:
$ mysql -h 192.168.0.200 -u root -p xfe356
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.5.9 MySQL Community Server (GPL)
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> USE mydatabase
Database changed
mysql> SET #texte = 'Le beau chien';
Query OK, 0 rows affected (0.00 sec)
mysql> SET #motif = 'e';
Query OK, 0 rows affected (0.00 sec)
mysql> SET #nb_motif = NULL;
Query OK, 0 rows affected (0.00 sec)
mysql> SET #positions = NULL;
Query OK, 0 rows affected (0.00 sec)
mysql> SET #out_nb_motif = 3;
Query OK, 0 rows affected (0.00 sec)
mysql> SET #out_positions = '2,5,12';
Query OK, 0 rows affected (0.00 sec)
mysql> CALL prc_liste_motif(#texte, #motif, #nb_motif, #positions);
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT #nb_motif = #out_nb_motif AND #positions = #out_positions;
+-----------------------------------------------------------+
| #nb_motif = #out_nb_motif AND #positions = #out_positions |
+-----------------------------------------------------------+
| 1 |
+-----------------------------------------------------------+
1 row in set (0.00 sec)
thanks in advance for your help !
Here is some pseudocode for one way you could test your database with RSpec:
describe "prc_liste_motif" do
before(:all) do
# Set up database connection here
end
describe "good values" do
context "Le beau chien" do
let(:texte) { "Le beau chien" }
# Set up other variables here
let(:results) { # call prc_liste_motif here }
it "has the correct out_nb_motif" do
out_nb_motif = # however you derive this from the results of the procedure
out_nb_motif.should == 3
end
it "has the correct out_positions" do
# test out_positions here
end
end
end
end
One thing I noticed in your sample manual test was how you are checking the results:
SELECT #nb_motif = #out_nb_motif AND #positions = #out_positions;
This will tell you whether or not those two values are correct, but if you get 0 results for this query, you do not immediately know which of the two values is incorrect and you do not know what the value you are getting instead is; getting that information requires more investigation.
By splitting up the checking for these two values into 2 RSpec tests, when the tests have finished running you can know if both are correct, if one is incorrect, or if both are incorrect. If one or both are incorrect, RSpec will also return a message for the failed test that says "Expected 3, got 4" which can help you debug faster.
As you add more tests for different inputs, I recommend refactoring the pseudocode I've given here to use shared_examples_for. The PragProg RSpec book that you're already reading is a great reference.
Cucumber's a natural-language BDD tool, which is designed to get non-technical stakeholders on board so that you can have conversations with them about what the system should do. It also lets you reuse steps quite easily - similar contexts, events and outcomes.
If you're writing a database, I think it's likely that your users, and the audience for that database, will be technical. There may also be limited opportunities for reusing steps, so Cucumber may not be the best tool. You're probably right about moving to something like RSpec instead. The English-language tools introduce a layer of abstraction and another aspect to maintenance which can be a pain in the neck, so I'd pick a tool which suits what you're doing, rather than starting with the tool and trying to fit your needs around it.
Once you've done that, you can either use ActiveRecord to create domain-object results from your queries, or you can just call the SQL directly. RSpec is just Ruby with some matchers. This forum might help you.
Something else you could do is to knock up a small application which actually uses your database. Not only will this ensure that your database is genuinely valuable; it will provide users with examples of how to use it. That won't be very difficult to do with Rails. If you go down this route, then you can use Cucumber with something like Webrat or Watir if you want to, because you'll be documenting the kind of things that other applications could use your database for at a higher level. Just make sure that
any live examples you provide go to
test data instead of production, and
that
if your little example app
suddenly turns into the real app
(which sometimes happens), you're in a position to spot that happening and take appropriate political and financial steps.
Java also has quite a lot of support for MySQL and you could use Hibernate instead of ActiveRecord, but I think the maintenance costs will be much less in Ruby.