Distribute records on different MySQL databases - MySQL Proxy alternative - mysql

My scenario is the following:
Right now I am using one big MySQL database with multiple tables to store user data. Many tables contain auto increment columns.
I would like to split this into 2 or more databases. The distribution should be done by user_id and is determined (cannot be randomized). E.g. user 1 and 2 should be on database1, user 3 on database2, user 4 on database3.
Since I don't want to change my whole frontend, I would like to still use one db adapter and kind of add a layer between the query generation (frontend) and the query execution (on the right database). This layer should distribute the queries to the right database based on the user_id.
I have found MySQL Proxy which sounds exactly like what I need. Unfortunately, it's in alpha and not recommended to be used in a production environment.
For php there is MySQL Native Driver Plugin API which sounds promising but then I need a layer that supports at least php and java.
Is there any other way I can achieve my objectives? Thanks!

This site seems to offer the service you're looking for (for a price).
http://www.sqlparser.com/
It lets you parse and modify queries and results. However what you're looking to do seems like it will only require a couple lines of code to distinguish between different user id's, so even though mysql-proxy is still in alpha your needs are simple enough that I would just use the proxy.
Alternatively, you could user whatever server-side language you're using to grab their user.id info, and then create a mysql connection to the appropriate database based on that info. Here's some php I scrabbled together which in spirit does what I think you're looking to do.
</php
// grab user.id from wherever you store it
$userID = get_user_id($clientUserName);
$userpass = get_user_pass($clientUserName);
if ($userID % 4 == 0) { // every 4th user
$db = new mysqli('localhost', $clientUserName, $userPass, 'db4');
}
else if ($userID % 3 == 0) { // every 3th user
$db = new mysqli('localhost', $clientUserName, $userPass, 'db3');
}
else if ($userID % 2 == 0) { // every 2nd user
$db = new mysqli('localhost', $clientUserName, $userPass, 'db2');
}
else // every other user
$db = new mysqli('localhost', $clientUserName, $userPass, 'db1');
}
$db->query('SELECT * FROM ...;');
?>

Related

Is there a way to store database modifications with a versioning feature (for eventual versions comparaison)?

I'm working on a project where users could upload excel files into a MySQL database. Those files are the main source of our data as they come directly from the contractors working with the company. They contain a large number of rows (23000 on average for each file) and 100 columns for each row!
The problem I am facing currently is that the same file could be changed by someone (either the contractor or the company) and when re-uploading it, my system should detect changes, update the actual data, and save the action (The fact that the cell went from a value to another value :: oldValue -> newValue) so we can go back and run a versions comparison (e.g 3 re-uploads === 3 versions). (oldValue Version1 VS newValue Version5)
I developed a tiny mechanism for saving the changes => I have a table to save Imports data (each time a user import a file a new row will be inserted in this table) and another table for saving the actual changes
Versioning data
I save the id of the row that have some changes, as well as the id and the table where the actual data was modified (Uploading a file results in a insertion in multiple tables, so whenever a change occurs, I need to know in which table that happened). I also save the new value and the old value which is gonna help me with restoring the "archives data".
To restore a version : SELECT * FROM 'Archive' WHERE idImport = ${versionNumber}
To restore a version for one row : SELECT * FROM 'Archive' WHERE idImport = ${versionNumber} and rowId = ${rowId}
To restore all version for one row : SELECT * FROM 'Archive' WHERE rowId = ${rowId}
To restore version for one table : SELECT * FROM 'Archine' WHERE tableName = ${table}
Etc.
Now with this structure, I'm struggling to restore a version or to run a comparaison between two versions, which makes think that I've came up with a wrong approach since it makes it hard to do the job! I am trying to know if anyone had done this before or what a good approach would look like?
Cases when things get really messy :
The rows that have changed in a version might not have changed in the other version (I am working on a time machine to search in other versions when this happens)
The rows have changed in both versions but not the same fields. (Say we have a user table, the data of the user with id 15 have changed in 2nd and 5th upload, great! Now for the second version only the name was changed, but for the fifth version his address was changed! When comparing these two versions, we will run into a problem constrcuting our data array. name went from "some"-> NULL (Name was never null. No name changes in 5th version) and address went from NULL -> "some' is which obviously wrong).
My actual approach (php)
<?php
//Join records sets and Compare them
foreach ($firstRecord as $frecord) {
//Retrieve first record fields that have changed
$fFields = $frecord->fieldName;
//Check if the same record have changed in the second version as well
$sId = array_search($frecord->idRecord, $secondRecord);
if($sId) {
$srecord = $secondRecord[$sId];
//Retrieve straversee fields that have changed
$sFields = $srecord->fieldName;
//Compare the two records fields
foreach ($fFields as $fField) {
$sfId = array_search($fField, $sFields);
//The same field for the same record was changed in both version (perfect case)
if($sfId) {
$sField = $sFields[$sfId];
$deltaRow[$fField]["oldValue"] = $frecord->deltaValue;
$deltaRow[$fField]["newValue"] = $srecord->deltaValue;
//Delete the checked field from the second version traversee to avoid re-checking
unset($sField[$sfId]);
}
//The changed field in V1 was not found in V2 -> Lookup for a value
else {
$deltaRow[$fField]["oldValue"] = $frecord->deltaValue;
$deltaRow[$fField]["newValue"] = $this->valueLookUp();
}
}
$dataArray[] = $deltaRow;
//Delete the checked record from the second version set to avoid re-checking
unset($secondRecord[$srecord]);
}
I don't know how to deal with that, as I said I m working on a value lookup algorithm so when no data found in a version I will try to find it in the versions between theses two so I can construct my data array. I would be very happy if anyone could give some hints, ideas, improvements so I can go futher with that.
Thank you!
Is there a way to store database modifications with a versioning feature (for eventual versions comparaison [sic!])?
What constitutes versioning depends on the database itself and how you make use of it.
As far as a relational database is concerned (e.g. MariaDB), this boils down to the so called Normal Form which is in numbers.
On Database Normalization: 5th Normal Form and Beyond you can find the following guidance:
Beyond 5th normal form you enter the heady realms of domain key normal form, a kind of theoretical ideal. Its practical use to a database designer os [sic!] similar to that of infinity to a bookkeeper - i.e. it exists in theory but is not going to be used in practice. Even the most demanding owner is not going to expect that of the bookkeeper!
One strategy to step into these realms is to reach the 5th normal form first (do this just in theory, by going through all the normal forms, and study database normalization).
Additionally you can construe versioning outside and additional to the database itself, e.g. by creating your own versioning system. Reading about what you can do with normalization will help you to find better ways to decide on how to structure and handle the database data for your versioning needs.
However, as written it depends on what you want and need. So no straight forward "code" answer can be given to such a general question.

WordPress - MySQL Auto Increment User ID

I am looking to run two instances of WordPress which sync on a regular basis.
The way I am going to do this is to have the auto increment offset on each different.
So for example on instance one the user IDs would be 1,3,5,7 etc. And on instance two the user IDs would be 2,4,6,8 etc.
Within WordPress there is the user function wp_insert_user which I would like to add a filter to for the database insert of a new user.
The issue is I cannot see a filter directly before or relating to the insert. So I am stuck on how I would achieve this.
The function is found in wp-includes\user.php file on line 1856 and within that on line 2060 is the insert statement for a new user:
$wpdb->insert( $wpdb->users, $data + compact( 'user_login' ) );
$user_id = (int) $wpdb->insert_id;
Could someone assist please? I just need pointing in the right direction as like I say I am not sure of the direction I need to take but want to ensure that the code I use is per the WordPress framework.
Configure this in MySQL directly. I assume you are going to sync all tables, in which case you might want to always increment by two. You can change the global auto_increment interval with auto_increment_increment:
SET GLOBAL auto_increment_increment=2;
Then configure each instance to have a different auto_increment_offset:
SET GLOBAL auto_increment_offset=1;
SET GLOBAL auto_increment_offset=2;

How to handle concurrency in Propel?

I'm new to the world of web & databases, and what I don't understand is how usually web application programmers synchronize database information when using ORM.
Let's say that we have this code:
$q = new AuthorQuery();
$firstAuthor = $q->findPK(1);
//Now someone changes the value of "LastViewedBy" in database,
but our PHP code still thinks that the value is "Me"
if( $q->getLastViewedBy() === "Me" )
{
//Do some stuff...
//Change information that affect the flow of this method
$q->save();
}
It's a classic synchronization problem (let's say that 10,000 people are executing this script at the same time).
How do I solve it when using Propel and MySQL (InnoDB)?

Convert Plain password into .NET Membership HASH password in T-SQL

We have a system that stores the username/password as plain text. I have been asked to covert this to Membership.
I'd like a SQL function to convert the plain text password into .NET Membership Hashed password. The only HashBytes function I found doesn't even come close to what I see in the .NET Membership table.
I am desperate. Please help.
I learnt that you cannot generate the same HASH algorithm in T-SQL and the best way is to simple read your basic table to the ASP.net side, then call the Membership Script to insert users. This cuts down on time as well.
DataSet dtData = DivDatabase.ExecuteSQLString("SELECT * FROM Users");
foreach (DataRow row in dtData.Tables[0].Rows)
{
MembershipUser mUser = null;
mUser = Membership.GetUser(row["Username"].ToString());
if (mUser == null)
{
mUser = Membership.CreateUser(row["Username"].ToString(), row["Password"].ToString(), row["Email"].ToString() );
}
}
I first check if the userName is not in the system already. This is because I had duplicating usernames and I wanted to eliminate that. For additional information from the old table that doesn't exist in the Membership tables, we agreed with the senior management that we are not going to use the Membership Profile as it's not easy to write queries against this. But I have added sample code for reference.
var profile = System.Web.Profile.ProfileBase.Create(row["Username"].ToString());
profile.SetPropertyValue("FirstName", row["FirstName"].ToString());
profile.SetPropertyValue("LastName", row["LastName"].ToString());
profile.Save();
I hope you find this useful.

how to simply handle a (very) short mysql replication lag

I have an application (php) running with MySQL 5.5 (1 master and 1 slave)
I use to dispatch read/write on master / slave.
When I create a new record (a user or something like that) I write it on the master and when I reload the page, I load it from the slave.
Example:
...
if ($_GET['id'])
{
#Load user
$user = $sql->load('user', $_GET['id']);
if ($user == false)
{
throw exception('User not found');
}
}
else if ($_POST['create]')
{
#Create a new user
$user_id = $sql->insert('user', $_POST);
$mvc->reload('?id=' . $user_id);
exit();
}
...
But when the master is really performant (quick insert) and the replication is not (lag = 0.3 - 1 sec), the reload will not work...
What are the best practice to handle that
Some solutions:
Database optimisation for reducing the lag (very difficult)
sleep(1) before reading or after writing ... not very elegant
First, you need to define if your application can work with a lag or not.
If not, then you need to ensure that the data you want to fetch are already available on slave. For example fetch the last id from slave and compare it with id you are about to fetch; or try slave first and if row is not there, fallback to master (but that will overload the master with requests for new data).
Usually web application can work with stale data. There is no problem if other visitors will see a new post 10 seconds later. But as you mentioned, it is bad if author of the post doesn't see it immediately. So you can act differently based on the data/reason you are fetching (for example cache the info about recent post in session and in that case fetch from master)