I have a master and slave database. To balance some of the load, I want all of my INSERT/UPDATE queries to go to the Master and all of my SELECT queries to go to the slave (since you cannot write to a slave without messing it up). I have the databases defined in the database.php. This is how I am trying to do the switching:
Inside the AppModel.php beforeFind(), I am setting:
$this->useDBConfig = 'readOnly';
Inside the beforeSave(), I am setting:
$this->useDBConfig = 'writeOnly';
However, after looking deeper I am wondering if the useDBConfig is a one-to-one relationship model-to-database. Meaning, maybe useDBConfig is only for pointing a single model to a single database, because this does not always work like I would expect. Sometimes an INSERT / UPDATE will attempt to run on a readOnly with readOnly permissions and fails.
If this is the case, how to I change my CakePHP application to send the requests to the appropriate databases? Should I be doing this a different way?
Basically, you override all of the save methods (save, saveField, saveAssociated, saveAll, UpdateAll, delete, deleteAll, etc) in the AppModel and encapsulate the parent call between two setDatasource calls.
Related
I am using TypeORM with MySQL and am setting up automatic auditing of all columns and database tables via MySQL Triggers - not TypeORM's "Logger" feature (unless you have some extra info)...
Without getting bogged down, the MySQL Triggers approach works very well and means no app-side code is required.
The problem: I cannot provide MySQL queries with the logged in app user's ID in a way that does not require we apply it in every query created in this app. We do have a central "CRUD" class, but that is for generic CRUD, so our more "specialist" queries would require special treatment - undesired.
Each of our tables has an int field "editedBy" where we would like to update with the user ID who edited the row (by using our app).
Question: Is there a way to intercept all non-read queries in TypeORM (regardless if its active record or query builder) and be able to update a column in the affected tables ('editedBy' int field)?
This would allow our Triggers solution to be complete.
P.S. I tried out TypeORM's custom logging function:
... typeorm createConnection({ ....
logger: new MyCustomLogger()
... });
class MyCustomLogger { // 'extend' has issue - works without anyway: extends Logger {
logQuery(query, parameters, somethingelse) // WORKS
{ ... }
logQuery does appear to fire before the query (I think) is sent to MySQL, but I cannot find a way how to extract the "Json-like" javascript object out of this, to modify each table's "editedBy". It would be great if there was a way to find all tables within this function and adjust editedBy. Happy to try other options... that don't entail updating the many files we have containing database calls.
Thanks
IMHO it should not be correct to use the logging feature of TypeOrm to modify your queries, it is very dangerous even if it would work with a bit of effort.
If you want to manage the way the upsert queries are done in TypeOrm, the best practice is to use custom repositories and then always calling it (not spawning vanilla repositories aftewards like in entityManager.getRepository(Specialist), instead use yours with entityManager.getCustomRepository(SpecialistRepository)).
The official documentation on the subject should help you a lot: https://github.com/typeorm/typeorm/blob/master/docs/custom-repository.md
Then in your custom repository you can override the save method and add whatever you want. Your code will be explicit and a good advantage is that it does not apply to every entity so if you have other different cases when you want to save differently, you are not stuck (you can also add custom save methods).
If you want to generalize the processing of the save methods, you can create an abstract repository to extend TypeOrm repository that you can then extend with your custom repository, in it you can add your custom code so that you don't end up copying it in every custom repository.
SpecialistRepository<Specialist> -> CustomSaveRepository<T> -> Repository<T>
I used a combination of https://github.com/skonves/express-http-context node module to pass user ID to TypeORM's Event Subscribers feature to make the update to data about to be submitted to DB: https://github.com/typeorm/typeorm/blob/master/sample/sample5-subscribers/subscriber/EverythingSubscriber.ts
Is there a way to tell sqlalchemy to log queries, but only those that change the state of the database (in particular INSERT, UPDATE and DELETE)?
I would like to keep a tally of all the changes made to the content of a mostly static database, but I do not want every web access to it be logged.
You can use SQLAlchemy-Continuum for this..its pretty cool...see the documentation to use - https://sqlalchemy-continuum.readthedocs.io/en/latest/
You can track all modification of your database with this.
Right now Play! automatically adds new tables to my mySQL database if I manually delete them. I remember reading a while back that it was possible to make play update the tables (without me needing to delete them first) when the models are changed.
I wasn't able to find anything with google, does anyone know how I can activate this? My biggest problems are the constraints that JPA is adding, they make it quite difficult do delete tables.
The way hibernate/play manages the database on Model changes is via the jpa.ddl property in your application.conf. If you read the file it states.
# Specify the ddl generation pattern to use. Set to none to disable it
# (default to update in DEV mode, and none in PROD mode):
# jpa.ddl=update
The options that I know about are
jpa.ddl=update - This simply updates the tables when a model changes
jpa.ddl=create-drop - This deletes the tables and recreates on model changes
jpa.ddl=validate -Just checks the schema, but does not make any changes
jpa.ddl=none - Does nothing
You can read more about this on the Hibernate site under the first property autoGenerateSchema
I would like to implement a custom database initialization strategy so that I can:
generate the database if not exists
if model change create only new tables
if model change create only new fields without dropping the table and losing the data.
Thanks in advance
You need to implement IDatabaseInitializer interface.
Eg
public class MyInitializer : IDatabaseInitializer<MyDbContext>
{
public void InitializeDatabase(MyDbContext context)
{
//your logic here
}
}
And then set your initializer at your application startup
Database.SetInitializer<ProductCatalog>(new MyInitializer());
Here's an example
You will have to manually execute commands to alter the database.
context.ObjectContext.ExecuteStoreCommand("ALTER TABLE dbo.MyTable ADD NewColumn VARCHAR(20) NULL");
You can use a tool like SQL Compare to script changes.
There is a reason why this doesn't exist yet. It is very complex and moreover IDatabaseInitializer interface is not very prepared for such that (there is no way to make such initialization database agnostic). Your question is "too broad" to be answered to your satisfaction. With your reaction to #Eranga's correct answer you simply expect that somebody will tell you step by step how to do that but we will not - that would mean we will write the initializer for you.
What you need to do what you want?
You must have very good knowledge of SQL Server. You must know how does SQL server store information about database, tables, columns and relations = you must understand sys views and you must know how to query them to get data about current database structure.
You must have very good knowledge of EF. You must know how does EF store mapping information. You must be able to explore metadata get information about expected tables, columns and relations.
Once you have old database description and new database description you must be able to write a code which will correctly explore changes and create SQL DDL commands for changing your database. Even this look like the simplest part of the whole process this is actually the hardest one because there are many other internal rules in SQL server which cannot be violated by your commands. Sometimes you really need to drop table to make your changes and if you don't want to lose data you must first push them to temporary table and after recreating table you must push them back. Sometimes you are doing changes in constraints which can require temporarily turning constrains off, etc. There is good reason why tools which do this on SQL level (comparing two databases) are probably all commercial.
Even ADO.NET team doesn't implemented this and they will not implement it in the future. Instead they are working on something called migrations.
Edit:
That is true that ObjectContext can return you script for database creation - that is exactly what default initializers are using. But how it could help you? Are you going to parse that script to see what changed? Are you going to execute that script in another connection to use the same code as for current database to see its structure?
Yes you can create a new database, move data from the old database to a new one, delete the old one and rename a new one but that is the most stupid solution you can ever imagine and no database administrator will ever allow that. Even this solution still requires analysis of changes to create correct data transfer scripts.
Automatic upgrade is a wrong way. You should always prepare upgrade script manually with help of some tools, test it and after that execute it manually or as part of some installation script / package. You must also backup your database before you are going to do any changes.
The best way to achieve this is probably with migrations:
http://nuget.org/List/Packages/EntityFramework.SqlMigrations
Good blog posts here and here.
I am very new to this and a good friend is in a bind. I am at my wits end. I have used gui's like navicat and sqlyog to do this but, only manually.
His band info data (schedules and whatnot) is in a MYSQL database on a server (admin server).
I am putting together a basic site for him written in Perl that grabs data from a database that resides on my server (public server) and displays schedule info, previous gig newsletters and some fan interaction.
He uses an administrative interface, which he likes and desires to keep, to manage the data on the admin server.
The admin server db has a bunch of tables and even table data the public db does not need.
So, I created tables on the public side that only contain relevant data.
I basically used a gui to export the data, then insert to the public side whenever he made updates to the admin db (copy and paste).
(FYI I am using DBI module to access the data in/via my public db perl script.)
I could access the admin server directly to grab only the data I need but, the whole purpose of this is to "mirror" the data not access the admin server on every query. Also, some tables are THOUSANDS of rows and parsing every row in a loop seemed too "bulky" to me. There is however a "time" column which could be utilized to compare to.
I cannot "sync" due to the fact that the structures are different, I only need the relevant table data from only three tables.
SO...... I desire to automate!
I read "copy" was a fast way but, my findings in how to implement were too advanced for my level.
I do not have the luxury of placing a script on the admin server to notify when there was an update.
1- I would like to set up a script to check a table to see if a row was updated or added on the admin servers db.
I would then desire to update or insert the new or changed data to the public servers db.
This "check" could be set up in a cron job I guess or triggered when a specific page loads on the public side. (the same sub routine called by the cron I would assume).
This data does not need to be "real time" but, if he updates something it would be nice to have it appear as quickly as possible.
I have done much reading, module research and experimenting but, here I am again at stackoverflow where I always get great advice and examples.
Much of the terminology is still quite over my head so verbose examples with explanations really help me learn quicker.
Thanks in advance.
The two terms you are looking for are either "replication" or "ETL".
First, replication approach.
Let's assume your admin server has tables T1, T2, T3 and your public server has tables TP1, TP2.
So, what you want to do (since you have different table structres as you said) is:
Take the tables from public server, and create exact copies of those tables on the admin server (TP1 and TP2).
Create a trigger on the admin server's original tables to populate the data from T1/T2/T3 into admin server's copy of TP1/TP2.
You will also need to do initial data population from T1/T2/T3 into admin server's copy of TP1/TP2. Duh.
Set up the "replication" from admin server's TP1/TP2 to public server's TP1/TP2
A different approach is to write a program (such programs are called ETL - Extract-Transform-Load) which will extract the data from T1/T2/T3 on admin server (the "E" part of "ETL"), massage the data into format suitable for loading into TP1/TP2 tables (the "T" part of "ETL"), transfer (via ftp/scp/whatnot) those files to public server, and the second half of the program (the "L") part will load the files into the tables TP1/TP2 on public server. Both halfs of the program would be launched by cron or your scheduler of choice.
There's an article with a very good example of how to start building Perl/MySQL ETL: http://oreilly.com/pub/a/databases/2007/04/12/building-a-data-warehouse-with-mysql-and-perl.html?page=2
If you prefer not to build your own, here's a list of open source ETL systems, never used any of them so no opinions on their usability/quality: http://www.manageability.org/blog/stuff/open-source-etl
I think you've misunderstood ETL as a problem domain, which is complicated, versus ETL as a one-off solution, which is often not much harder than writing a report. Unless I've totally misunderstood your problem, you don't need a general ETL solution, you need a one-off solution that works on a handful of tables and a few thousand rows. ETL and Schema mapping sound scarier than they are for a single job. (The generalization, scaling, change-management, and OLTP-to-OLAP support of ETL are where it gets especially difficult.) If you can use Perl to write a report out of a SQL database, you probably know enough to handle the ETL involved here.
1- I would like to set up a script to check a table to see if a row was updated or added on the admin servers db. I would then desire to update or insert the new or changed data to the public servers db.
If every table you need to pull from has an update timestamp column, then your cron job includes some SELECT statements with WHERE clauses based on the last time the cron job ran to get only the updates. Tables without an update timestamp will probably need a full dump.
I'd use a one-to-one table mapping unless normalization was required... just simpler to my opinion. Why complicate it with "big" schema changes if you don't have to?
some tables are THOUSANDS of rows and parsing every row in a loop seemed too "bulky" to me.
Limit your queries to only the columns you need (and if there are no BLOBs or exceptionally big columns in what you need) a few thousand rows should not be a problem via DBI with a FETCHALL method. Loop all you want locally, just make as few trips to the remote database as possible.
If a row is has a newer date, update it. I will also have to check for new rows for insertion.
Each table needs one SELECT ... WHERE updated_timestamp_columnname > last_cron_run_timestamp. That result set will contain all rows with newer timestamps, which contains newly inserted rows (if the timestamp column behaves like I'd expect). For updating your local database, check out MySQL's ON DUPLICATE KEY UPDATE syntax... this will let you do it in one step.
... how to implement were too advanced for my level ...
Yes, I have actually done this already but, I have to manually update...
Some questions to help us understand your level... Are you hitting the database from the mysql client command-line or from a GUI? Have you gotten to the point where you've wrapped your SQL queries in Perl and DBI, yet?
If the two databases have different, you'll need an ETL solution to map from one schema to another.
If the schemas are the same, all you have to do is replicate the data from one to the other.
Why not just create identical structure on the 'slave' server to the master server. Then create a small table that keeps track of the last timestamp or id for the updated tables.
Then select from the master all rows changed since the last timestamp or greater than the id. Insert them into the matching table on the slave server.
You will need to be careful of updated rows. If a row on the master is updated but the timestamp doesn't change then how will you tell which rows to fetch? If that's not an issue the process is quite simple.
If it is an issue then you need to be more sophisticated, but without knowing the data structure and update mechanism its a goose chase to give pointers on it.
The script could be called by cron every so often to update the changes.
if the database structures must be different on the two servers then a simple translation step may need to be added, but most of the time that can be done within the sql select statement and maybe a join or two.