How do you run a MySQL query from a rake task? - mysql

I'm working on removing duplicates from a legacy database for a client of mine, and I've found a MySQL query to do just that. I want to create a Rake task to run the query for the production server. How would I do that?
MySQL query:
select * from community_event_users;
create table dups as
select distinct username, count(*)
from community_event_users group by username
having count(*) > 1;
delete community_event_users from community_event_users inner join dups
on community_event_users.username = dups.username;
insert into community_event_users select username from dups;

If you are on Rails and using ActiveRecord you can simply use:
ActiveRecord::Base.execute(my_sql)
ActiveRecord::Base.connection.execute(my_sql)
Where my_sql is your SQL string.

For Rails 5 and above you better use:
ApplicationRecord.connection.execute <<~END_OF_SQL
YOUR QUERY HERE
END_OF_SQL

Related

MySQL query with distinct and order by showing error 3065(HY000)

I am using linux mint in my machine and trying to execute a query. After the execution, an error is displayed
ERROR 3065 (HY000): Expression #1 of ORDER BY clause is not in SELECT list, references column 'shelterl_local.animal.changed' which is not in SELECT list; this is incompatible with DISTINCT
I added line
[mysqld]
sql-mode=''
in /etc/mysql/my.cnf file and restarted mysql. But still getting the same error. I referred many solutions but nothing worked for me. My query is
SELECT DISTINCT fs.etid AS etid FROM og_membership fs
LEFT OUTER JOIN node animal ON animal.nid = fs.etid LEFT OUTER JOIN
field_data_field_for_animal dfa ON dfa.field_for_animal_value = fs.etid
LEFT OUTER JOIN node pastid ON pastid.nid = dfa.entity_id WHERE ( (fs.gid =
464) OR
(animal.nid IN
(1196113,1211208,1218831,1243640,1254254,
1253603,1249890,1261729,1261282,1258378,1273745,1270760,
1279219,1276040,1276031,1275684,1288727,1289306,1300545,
1313770,1313761,1313755,1313746,1313330,1312388,1310673,
1309431,1315024,1333640,1328041,1323565,1327216,1330104,
1327786,1326810,1335812,1333094,1341309,1340358,1348088,
1351077,1351071,318214,1342698,1472755,1491527,1351652,1353488,
1507763,1342713)) )AND (fs.entity_type = 'node')
AND (animal.type = 'animal') AND (animal.status = 1) AND (pastid.title LIKE
'%A%')
ORDER BY animal.changed DESC LIMIT 0,300;
Is it possible to remove this error permanently and execute the query? Please help
Your initial query is equivalent to the following:
SELECT field1 AS f1
FROM table t
--Joins and conditions
GROUP BY field1
ORDER BY field2 DESC LIMIT 0,300
This can't make logical sense, because each value of field1 in the result set may have multiple values of field2 associated with it. The error you are seeing is MySQL's way of saying it can't figure out what you want to do. One workaround would be to sort on an aggregate of field2, e.g. try the following:
SELECT field1 AS f1
FROM table t
--Joins and conditions
GROUP BY field1
ORDER BY MAX(field2) DESC -- or AVG(field2), or MIN(field2), etc.
LIMIT 0,300
You need to actually select field2
SELECT DISTINCT field1 AS f1,
field2
FROM table t
--Joins and conditions
ORDER BY field2 DESC LIMIT 0,300
UPDATE:
I know sometimes theres errors when using DISTINCT and ORDER BY in the same query. To fix this i would give the animal.changed section an ALIAS such as [animal.changed] and then if you ORDER BY [animal.changed] this should not error. At least i know this definitely works in SQL server
E.G
SELECT DISTINCT animal.changed AS [animal.changed]
FROM .....
ORDER BY [animal.changed]
This is very basic but would allow you to use DISTINCT and ORDER BY in the same query which is the current error you're getting.
If you select a column then sorting can only be done for this column. If you need to sort by another column, then necessarily include this column in the selection.
//wrong.....
$sql="Select DISTINCT Rubrika from tmp3 order by View desc limit 3";
$res_r=mysqli_query($Link, $sql);
//RIGHT!!!
$sql="Select DISTINCT Rubrika, View from tmp3 order by View desc limit 3";
$res_r=mysqli_query($Link, $sql);
I also encountered the same issue and abled to resolve it. Since I'm using Windows I'll post fix for windows but similar fix will work for linux as well.
This is happening because ONLY_FULL_GROUP_BY mode is enabled by mysql. You can disable this by modifying my.ini file. (In linux this would be my.cnf).
Open my.ini file.
This file might be located in a hidden directory ProgramData.
C:\ProgramData\MySQL\MySQL Server 8.0
Find following config which is placed under [mysqld] section.
sql-mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"
Remove ONLY_FULL_GROUP_BY from here
sql-mode="STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"
Restart mysql server
net stop MYSQL80
net start MYSQL80

Different results for query on mysql 5.6 and 5.7

I have a laravel application that was running an elqouent query with a whereHas method. The method was not returning any data on a virtual machine running mysql 5.7 but was working on 5.6. I got the raw query out of eloquent, here's what it is:
select * from `threads` where `id` = 5053 and exists (select * from `users` inner join `threads_users` on `users`.`id` = `threads_users`.`user_id` where `threads_users`.`thread_id` = `threads`.`id` and `user_id` = 296) limit 1
When I run this sql in mysql 5.6 it will return a single row but when I run it on mysql 5.7 it returns nothing. Is there some feature that was removed between these versions that would break this query?

How do I write an AREL UpdateManager query for MySQL that uses a subquery

I am tying to run an update query with a subquery against a MySQL database using ruby. I am using ruby 1.9.3 and rails 4.1.
The query I am trying to create is as below:
UPDATE `items`
SET
`items`.`status_id` = 12
WHERE
`items`.`id` IN (SELECT DISTINCT
`items`.`id`
FROM
`items`
LEFT OUTER JOIN
`statuses` ON `items`.`status_id` = `statuses`.`id`
LEFT OUTER JOIN
`resources` ON `items`.`resource_id` = `resources`.`id`
WHERE
`statuses`.`title` LIKE 'On Loan'
AND `items`.`duedate` < '2015-04-24'
AND `items`.`return_date` IS NULL
ORDER BY `items`.`duedate`)
I can produce this query in ruby using AREL with the code shown below:
# Declare Arel objects
i = Item.arel_table
s = Status.arel_table
r = Resource.arel_table
# This is the AREL query that returns the data
overdues = i.project(i[:id]).
join(s, Arel::Nodes::OuterJoin).on(i[:status_id].eq(s[:id])).
join(r, Arel::Nodes::OuterJoin).on(i[:resource_id].eq(r[:id])).
where(s[:title].matches("On Loan").
and(i[:duedate].lt(DateTime.now.to_date)).
and(i[:return_date].eq(nil))
).
order(i[:duedate])
# Note: You can't chain distinct, otherwise "overdues" becomes a string with the value "DISTINCT".
overdues.distinct
# This creates the update...
u = Arel::UpdateManager.new i.engine
u.table(i)
u.set([[i[:status_id], 10]]).where(i[:id].in(overdues))
This does not work and returns an error message:
ActiveRecord::StatementInvalid: Mysql2::Error: You can't specify target table 'items' for update in FROM clause:
I tried using AR "update_all" but it produces the same SQL and hence the same error.
Item.where(i[:id].in(overdues)).update_all(:status_id => (Status.find_by(:title => "Overdue").id))
Having done some research I have found that you cannot run a update with a subquery that references the table you want to update in MySQL. I have seen a number of posts on this site and the wider internet that detail work arounds.
One suggestion says that the update should use a join instead of a sub query. Having looked at the code behind the update manager it has no "join" so I can't do that.
Another says run this in two parts but I can't see how to because AREL and AciveRecord both chain actions.
The only way I can see of doing this is by aliasing the table and adding an additional select (see below). This isn't great but it would be useful to see if it is possible to do.
UPDATE `items`
SET `status_id` = 10
WHERE `items`.`id` IN (
SELECT x.id
FROM
(SELECT DISTINCT `items`.`id`
FROM `items`
LEFT OUTER JOIN `statuses` ON `items`.`status_id` = `statuses`.`id`
LEFT OUTER JOIN `resources` ON `items`.`resource_id` = `resources`.`id`
WHERE `statuses`.`title` LIKE 'On Loan'
AND `items`.`duedate` < '2015-04-24'
AND `items`.`return_date` IS NULL
ORDER BY `items`.`duedate`) x
);
If I can't get this to work I could adopt two other approaches:
1) I could just hard-code the SQL but I want to use ActiveRecord and reference the models to keep it database agnostic.
2) The other way is to return an instance of all the records and loop through them doing individual updates. This will have a performance issue but I can accept this because its a background job that won't be updating more than a handful of records each day.
Update
I have the AREL query below that produces the subquery in the format I need.
x = Arel::Table.new('x')
overdues = Item.select(x[:id]).from(
Item.select(Item.arel_table[:id]).where(
Status.arel_table[:title].matches("On Loan").and(
Item.arel_table[:duedate].lt(DateTime.now.to_date).and(
Item.arel_table[:return_date].eq(nil))
)
).joins(
Item.arel_table.join(Status.arel_table, Arel::Nodes::OuterJoin).on(
Item.arel_table[:status_id].eq(Status.arel_table[:id])
).join_sources
).joins(
Item.arel_table.join(Resource.arel_table, Arel::Nodes::OuterJoin).on(
Item.arel_table[:resource_id].eq(Resource.arel_table[:id])
).join_sources
).order(Item.arel_table[:duedate]).uniq.as('x')
)
Sadly it returns an error when I use it in my update statement.
TypeError: Cannot visit Item::ActiveRecord_Relation
Having revisited this question I am at the conclusion that it's not possible to do this because of a limitation with MySQL:
ActiveRecord::StatementInvalid: Mysql2::Error: You can't specify target table 'items' for update in FROM clause:
It should be possible to do with other databases (although I haven't tested that).
I could create a temporary table, which is the copy of the original table, reference that and then drop the temporary table like this post suggests:
http://richtextblog.blogspot.co.uk/2007/09/mysql-temporary-tables-and-rails.html. That seems a lot of overhead to do a simple subquery.
What I am going to do is find all the ID's and loop through them and update the records that way (using a simple find and update). This has an overhead but it should only be updating a handful of records each run (no more than 100). The update will be running as a scheduled job outside user working hours so it won't impact performance.
I still find it bizarre that in all other flavours of SQL I have never encountered this problem before. Still you live and learn.
UPDATE:
Since updating my version of MySQL the select statement now works. I had to take out the order by for it to work.
ORDER BY `items`.`duedate`
I am now using version: 5.7.19.

mysql: Java api to call mysql metadata service

Is there any java api to call mysql metadata service? The things I am particularly interested in is getting schema of the table using api not modifying the schema of the table.
The best source for getting table metadata is MySQL itself. Use the INFORMATION_SCHEMA tables/views in MySQL to get this data. You can execute the queries and read the results set back like a normal query.
For table information:
SELECT * FROM INFORMATION_SCHEMA.TABLES
For columns:
SELECT * FROM INFORMATION_SCHEMA.COLUMNS
You can use this form for indexes on InnoDB:
SELECT t.name AS `Table`,
i.name AS `Index`,
GROUP_CONCAT(f.name ORDER BY f.pos) AS `Columns`
FROM information_schema.innodb_sys_tables t
JOIN information_schema.innodb_sys_indexes i USING (table_id)
JOIN information_schema.innodb_sys_fields f USING (index_id)
WHERE t.schema = 'sakila'
GROUP BY 1,2;

mySQL Nested Query Syntax

I am trying to use a nested query approach to build a query-on-query for my mySQL database and failing to correctly generate output. I am able to import my table into Microsoft Access and build Query1 and then build Query2 on Query1 to get the correct output I'm looking for so I feel like I'm close, I just can't get the right syntax to get the output I'm looking for using a mySQL query approach.
Query1, here is the SQL statement from Access for Query1.
SELECT DISTINCT MediaBuys.DistrictID, MediaBuys.SpenderID, MediaBuys.PG, MediaBuys.SupportType, MediaBuys.PriSupportType
FROM MediaBuys
WHERE MediaBuys.PG ="P";
Query2, if I have built Query1 in Access as above and I run this SQL statement in Access as a separate query built on the first I can generate the output I'm looking for.
SELECT Query1.DistrictID, Query1.SpenderID, Query1.PG, Query1.SupportType, Query1.PriSupportType, Count(Query1.SupportType) AS CountOfSupportType
FROM Query1 INNER JOIN Query1 AS Query1_1 ON Query1.PG = Query1_1.PG AND Query1.SpenderID = Query1_1.SpenderID AND Query1.DistrictID = Query1_1.DistrictID
GROUP BY Query1.DistrictID, Query1.SpenderID, Query1.PG, Query1.SupportType, Query1.PriSupportType
HAVING Count(Query1.SupportType) > 1;
I'd like to be able to produce the same output from a query in mySQL. Since I have the SQL statements of these two queries I feel like this should be doable, I've attempted to build a nested query in a number of different ways and each attempt fails, it seems I can't put together the correct syntax. The most common error I receive is "Error Code: 1146. Table 'Query1' doesn't exist".
Is this doable in mySQL and if so can anyone help me with the correct syntax?
Just like you created the query Query1 in Access, create a view View1 in MySql:
CREATE VIEW View1 AS
SELECT DISTINCT DistrictID, SpenderID, PG, SupportType, PriSupportType
FROM MediaBuys
WHERE PG ='P';
and your query will be:
SELECT
View1.DistrictID, View1.SpenderID, View1.PG, View1.SupportType, View1.PriSupportType,
Count(View1.SupportType) AS CountOfSupportType
FROM View1 INNER JOIN View1 AS View1_1
ON View1.PG = View1_1.PG AND View1.SpenderID = View1_1.SpenderID
AND View1.DistrictID = View1_1.DistrictID
GROUP BY View1.DistrictID, View1.SpenderID, View1.PG, View1.SupportType, View1.PriSupportType
HAVING Count(View1.SupportType) > 1;