I have some serious Performance Problems and i found out that it is because the lack of Indexes in MySQL. So i added an Index to the Table Definition of the Entity:
#ORM\Table(
name="test",
indexes={
#ORM\Index(name="test_idx", columns={"testfield"})
}
)
These lines are ok (hopefully) but resulting in simply nothing, When i run doctrine orm:validate-schema it says, that the Database is in Sync. When i add the Index manually to MySQL it says it is no longer in sync, and wants to drop the index. I am a little bit confused because the CLI-Tool does not add the Index (but drops it, if it exists), and i am not getting any error message? What ist wrong?
Should work this way, though, the Index-Support ist broken at the moment. Have a look at:
https://github.com/doctrine/DoctrineORMModule/issues/368
Related
In my MySQL database I have a user table. I need to perform search as you type with typo over the user name field. There are few very old question on this topic. I tested the builtin full text search of mysql but it didn't work as expected (it does not handle typo) [I knew but I tried anyway].
What's my best option? I thought there should be an easy solution nowadays. I'm thinking about replicating the user table on elasticsearch and do the instant search from there, but I'd really like to avoid the syncronization nightmare that this will cause.
Thanks!!
You could use SOUNDEX for mysql. We have tried that but I can say that it does not work that well and it also makes the search a bit slow.
We Had a similar issue and switched to ES.
What we did is as follows:
Created a trigger for the table that will be synced to ES. The
trigger will write to a new table. The columns of such a table would
be:
IdToUpdate Operation DateTime IsSynced
The Operation would be create, update, delete. IsSynced will tell
whether the update is pushed to ES.
Then add a corn job that would query this table for all rows that will have issynced set to say '0', Add those ID's and operation to a Queue like RabbitMQ. And set the ISSynced to 1 for those ID's
The reason to use RabbitMQ is that it will make sure that the update is forwarded to ES. In case of failure we can always re-queue the the object.
Write a consumer to get the objects from the queue and update ES.
Apart from this you will also have to create a utility that will create an ES index from the database for first time use.
And you can also look at Fuzzy Search of ES that will handle typo's
Also Completion suggester which also supports fuzzy search.
I have a text field in my database and a index on it for the first 10 characters. How do I specify that in my Doctrine Entity?
I can't find any information about database specific options for indexes anywhere :/
This is my "partial" MySQL create statement:
KEY `sourceaddr_index` (`sourceaddr`(10)),
And this is my #Index in doctrine:
#ORM\Index(name="sourceaddr_index", columns={"sourceaddr"}, options={}),
This dosnt interfere with the regular use, but I noticed the problem when deploying development to a new laptop, and creating the database based on my entities...
Any help would be appreciated :)
Possible since Doctrine 2.9, see: https://github.com/doctrine/dbal/pull/2412
#Index(name="slug", columns={"slug"}, options={"lengths": {191}})
Unfortunately, Doctrine seem to be very picky with whitespace location, so e.g. update --dump-sql yields:
DROP INDEX slug ON wp_terms;
CREATE INDEX slug ON wp_terms (slug(191));
and even if you execute those, they messages will stay there (tested with MariaDB 10.3.14).
I've had very good luck naming the index in Doctrine, after manually creating it in MySQL. It's not pretty or elegant, and it's prone to cause errors moving from dev to production if you forget to recreate the index. But, Doctrine seems to understand it respect it.
In my entity, I have the following definition. Doctrine ignores the length option - it's wishful thinking on my part.
/**
* Field
*
* #ORM\Table(name="field", indexes={
* #ORM\Index(name="field_value_bt", columns={"value"}, options={"length": 100})
* })
And in MySQL, I execute
CREATE INDEX field_value_bt ON field (value(100))
As far as I've seen, Doctrine just leaves the index alone so long as it's named the same.
In short: you can't set this within Doctrine. Doctrine's ORM is specifically focused on cross vendor compatability and the type of index you're describing, though supported in many modern RDBMS, is somewhat outside the scope of Doctrine to handle.
Unfortunately there isn't an easy way around this if you use Doctrine's schema updater (in Symfony that would be php app/console doctrine:schema:update --force) as if you manually update the database, Doctrine will sometimes, regress that change to keep things in sync.
In instances where I've needed something like this I've just set up a fixture that sends the relevant ALTER TABLE statement via SQL. If you're going to be distributing your code (i.e. it may run on other/older databases) you can wrap the statement with a platform check to make sure.
It's not ideal but once your app/software stabilises, issues like this shouldn't happen all that often.
When upgrading a site for customer to a new server with MySQL 5 instead of MySQL 4, I ran into the usual bunch of problems when migrating the database, some of which simply make mysql complain about the SQL dump (those were the easiest to fix) while others required a few changes in the ASP code. Like using commas instead of JOINs isn't always viable in MySQL 5.
One problem is more annoying though. With MySQL 5, most ASP statements dealing with comparisons now fail with a Type Mismatch, such as :
IF RS("myfield") > 5 THEN
That especially happens if the SQL query involves SUMs. It works fine with MySQL 4.
The fix is not difficult: For the queries, a CAST to SIGNED fixes it, and for the comparison statements themselves, using Cint does the trick.
The problem is though, the site is huge and consists of hundreds of separate ASP files (the code is really bad!) and it is almost impossible to hunt down all these Type Mismatch errors until they show up.
So my question is:
Is there a more global way to fix this problem? I think of settings in the ODDC driver or MySQL itself etc.
Doing the change in the DB instead of the asp code could be a way to go. That way you dont have to change that many webpages. The question is if these fields are possible to change for your needs?
Did you also change database type? InnoDB or MyISAM, that could also change the way field are handled?
I use Kettle for some transformations and ran into a problem:
For one specific row, my DatabaseLookup step hangs. It just doesn't give a result. Trying to stop the transformation results in a never ending "Halting" for the lookup step.
The value given is nothing complicated at all, neither it is different from all other rows/values. It just won't continue.
Doing the same query in the database directly or in a different database tool (e.g. SQuirreL), it works.
I use Kettle/Spoon 4.1, the database is MySQL 5.5.10. It happens with Connector/J 5.1.14 and the one bundled with spoon.
The step initializes flawlessly (it even works for other rows) and I have no idea why it fails. No error message in the Spoon logs, nothing on the console/shell.
weird. Whats the table type? is it myisam? Does your transform also perform updates to the same table? maybe you are locking the table inadvertantly at the same time somehow?
Or maybe it's a mysql 5.5 thing.. But ive used this step extensively with mysql 5.0 and pdi 4.everything and it's always been fine... maybe post the transform?
I just found the culprit:
The lookup takes as a result the id field and gave it a new name, PERSON_ID. This FAILS in some cases! The resulting lookup/prepared statement was something like
select id as PERSON_ID FROM table WHERE ...
SOLUTION:
Don't use underscore in the "New name" for the field! With a new name of PERSONID everything works flawlessly for ALL rows!
Stupid error ...
I've recently been working with a MySQL database, and using MySQL workbench to design the Database.
When I use the export to SQL function, so I can actually get the layout in to the Database, I get:
"Error 1005: Cannot create table"
This appears to be related to Foreign Keys in the create table statement.
Does anybody have a work around for this that doesn't involve taking the constraints out and putting them back in later? That's a less than ideal solution given the size of the database.
When you get this (and other errors out of the InnoDB engine) issue:
SHOW ENGINE INNODB STATUS;
It will give a more detailed reason why the operation couldn't be completed. Make sure to run that from something that'll allow you to scroll or copy the data, as the response is quite long.
I ran into this situation recently when I attempted (in InnoDB tables) to make a foreign key reference to a column that had a different data type.
MySQL 5.1 Documentation