im trying to detect and delete a line break out of a subject (called m.subject) mail information retrieved via CONCAT out of a mysql database.
That said, the linebreak may or may not occur in the subject and therefore must be detected.
My query looks like this:
mysql --default-character-set=utf8 -h $DB_HOST -D $TARGET -u $DB_USER -p$DB_PW -N -r -e "SELECT CONCAT(m.one,';',m.two,';',m.three,';',m.subject,';',m.four';',m.five,';',(SELECT CONCAT(special_one) FROM special_$SQL_TABLE WHERE msg_id = m.six ORDER BY time DESC LIMIT 1)) FROM mails_$SQL_TABLE m WHERE m.rtime BETWEEN $START AND $END AND m.seven = 1 AND m.eight IN (2);"
I tried to delete it afterwards, but getting in performance trouble due to several while operations on all lines already. Is there an easy way to detect and cut it directly via the CONCAT buildup? It is crucial to retrieve only one line after extraction for me.
Updating/changing the database is not an option for me, as I only want to read the current state.
Related
I'm trying to update multiple rows in a DB using a small script.
I need to update the rows based on some specific user_ids which I have in a list on Linux machine.
#! /bin/bash
mysql -u user-ppassword db -e "update device set in_use=0 where user_id in ()";
As you see above, the user_ids are in a file, let's say /opt/test/user_ids_txt.
How can I import them into this command?
This really depends on the format of user_ids_txt. If we assume it just happens to be in the correct syntax for your SQL in statement, the following will work:
#! /bin/bash
mysql -u user-ppassword db -e "update device set in_use=0 where user_id in ($(< /opt/test/user_ids_txt))";
The bash interpreter will substitute in the contents of the file. This can be dangerous for SQL queries, so I would echo out the command on the terminal to make sure it is correct before implementing it. You should be able to preview your SQL query by simply running the following on the command line:
echo "update device set in_use=0 where user_id in ($(< /opt/test/user_ids_txt))"
If your file is not in the SQL in syntax you will need to edit it (or a copy of it) before running your query. I would recommend something like sed for this.
Example
Let's say your file /opt/test/user_ids_txt is just a list of user_ids in the format:
aaa
bbb
ccc
You can use sed to edit this into the correct SQL syntax:
sed 's/^/\'/g; s/$/\'/g; 2,$s/^/,/g' /opt/test/user_ids_txt
The output of this command will be:
'aaa'
,'bbb'
,'ccc'
If you look at this sed command, you will see 3 separate commands separated by semicolons. The individual commands translate to:
1: Add ' to the beginning of every line
2: Add ' to the end of every line
3: Add , to the beginning of every line but the first
Note: If your ID's are strictly numeric, you only need the third command.
This would make your SQL query translate to:
update device set in_use=0 where user_id in ('aaa'
,'bbb'
,'ccc')
Rather than make a temporary file to store this, I would use a bash variable, and simply plug that into the query like this:
#! /bin/bash
in_statement="$(sed 's/^/\'/g; s/$/\'/g; 2,$s/^/,/g' /opt/test/user_ids_txt)"
mysql -u user-ppassword db -e "update device set in_use=0 where user_id in (${in_statement})";
All of the other variables that make this work are tested and working correctly so I'm obviously doing this wrong.
I have a bash script that first selects some mysql data and stores into a new variable.
Then it goes on to connect again and update the database.
title=$(mysql -u $user -p$pass -h $host dbname | SELECT post_title FROM wp_posts WHERE ID=$8);
mysql --host=$host --user=$user --password=$pass dbname <<EOF
UPDATE wp_my_music_lib SET title = "$title" WHERE track_id=${4}${6};
EOF
The title entry is always blank which says to me that the initial SELECT isn't working properly. It should also be noted that the data expected from the select result has white space and special chars in it ie :
Some Artist (10/10/13)
I thought quoting the var "$title" would fix any potential problems with gobbling but that isn't the issue here as I've tried selecting a single numerical object from a different column and that doesn't work either.
If I hard code the title var it works as expected.
1) Can you see what I'm doing wrong?
2) Is it possible to perform all of the above with one db connection instead as that would make more sense?
mysql | SELECT pipes the output of mysql to a command called SELECT, which is сertainly not what you want.
To execute a query via mysql and capture the output you can use this syntax:
title=$(mysql -B dbname <<< "SELECT post_title FROM wp_posts WHERE ID=$8")
You could also execute the SELECT in a subquery to avoid multiple calls to mysql:
mysql --host=$host --user=$user --password=$pass dbname <<EOF
UPDATE wp_my_music_lib SET title = (
SELECT post_title FROM wp_posts WHERE ID=$8)
WHERE track_id=${4}${6}
EOF
I need to run a MySQL script that, according to my benchmarking, should take over 14 hours to run. The script is updating every row in a 332715-row table:
UPDATE gene_set SET attribute_fk = (
SELECT id FROM attribute WHERE
gene_set.name_from_dataset <=> attribute.name_from_dataset AND
gene_set.id_from_dataset <=> attribute.id_from_dataset AND
gene_set.description_from_dataset <=> attribute.description_from_dataset AND
gene_set.url_from_dataset <=> attribute.url_from_dataset AND
gene_set.name_from_naming_authority <=> attribute.name_from_naming_authority AND
gene_set.id_from_naming_authority <=> attribute.id_from_naming_authority AND
gene_set.description_from_naming_authority <=> attribute.description_from_naming_authority AND
gene_set.url_from_naming_authority <=> attribute.url_from_naming_authority AND
gene_set.attribute_type_fk <=> attribute.attribute_type_fk AND
gene_set.naming_authority_fk <=> attribute.naming_authority_fk
);
(The script is unfortunate; I need to transfer all the data from gene_set to attribute, but first I must correctly set a foreign key to point to attribute).
I haven't been able to successfully run it using this command:
nohup mysql -h [host] -u [user] -p [database] < my_script.sql
For example, last night, it ran over 10 hours but then the ssh connection broke:
Write failed: Broken pipe
Is there any way to run this script in a way to better ensure that it finishes? I really don't care how long it takes (1 day vs 2 days doesn't really matter) so long as I know it will finish.
The quickest way might be to run it in a screen or tmux session.
Expanding on my comment, you're getting poor performance for a 350k record update statement. This is because you're setting based on the result of a sub query, and not updating as a set. Thus you're running the statement once for each row. Update as such:
UPDATE gene_set g JOIN attribute_fk a ON < all where clauses > SET g.attribute_fk = a.id.
This doesn't answer your question per se, but I'll be interested to know how much faster it runs.
Here is how i did it in past where I ran monolithic alter queries in remote server which take ages sometime :
mysql -h [host] -u [user] -p [database] < my_script.sql > result.log 2>&1 &
This way you don't need to wait for it as it will finish on its own time,You could customize and add select now() at start and end in your my_script.sql to find out how long it took if you interest .
Things also to consider if applicable
Why this query take this long, can we improve it(eg : disable key checks .. , offline prepare the data and update with a temp table ..
Can we break the query to run in batches
What is the impact on rest of the DB
etc
If you have ssh access to the server you could copy it over and run it there with the following lines:
#copy over to tmp dir
scp my_script.sql user#remoteHost:/tmp/
#execute script on remote host
ssh -t user#remoteHost "nohup mysql \
-h localhost -u [user] -p [database] < /tmp/my_script.sql &"
Maybe you can try to do 300k updates with frequent commits instead of one single huge update. Doing that inc ase anything failed at you will maintain the changes already applied.
with some dimacic sql you can get all the lines in one go, later copy the file to your server ...
I am storing output of MySQL query in a varible using shell scripting. The output of SQL query is in multiple rows. When I checked the count of the variable (which I think is an array), it is giving 1. My code snippet is as follows:
sessionLogin=`mysql -ugtsdbadmin -pgtsdbadmin -h$MYSQL_HOST -P$MYSQLPORT CMDB -e " select distinct SessionID div 100000 as 'MemberID' from SessionLogin where ClientIPAddr like '10.104%' and LoginTimestamp > 1426291200000000000 order by 1;"`
echo "${#sessionLogin[#]}"
How can I store the MySQL query output in an array in shell scripting?
You can loop over the output from mysql and append to an existing array. For example, in Bash 3.1+, a while loop with process substitution is one way to do it (please replace the mysql parameters with your actual command)
output=()
while read -r output_line; do
output+=("$output_line")
done < <(mysql -u user -ppass -hhost DB -e "query")
echo "There are ${#output[#]} lines returned"
Also take a look at the always excellent BashFaq
My script cycles through multiple environments and if a user is found, it prints out their ID. The desired output is as follows:
environment1
<user_id>
However, my script is currently working to print out every environment even if there isn't a user id, like follows:
environment1
<user_id>
environment2
environment3
environment4
<user_id>
I would like the code to print out "NO USER FOUND" if the user_id does not exist, rather than excluding the environment altogether.
The code below is what is being utilized:
#log it all
{
echo "environment"
sshpass -p $ldappw ssh $ldapuser#12.345.67.89 'mysql --skip-column-names -hhost -u user -ppassword database -e \
"select user_id from users where first like '"'%${first_name}%' and last like '%${last_name}%';"'"'
} > $log
Any help would be much appreciated for this issue. Unfortunately I'm just beginning to learn more advanced MySQL and Linux command line tools and am not quite proficient enough to know how to handle this problem.
may be
select ifnull(user_id,'NO USER FOUND') from users where first like '"'%${first_name}%' and last like '%${last_name}%';