Unable to understand this strange MySql client behavior - mysql

I'm not sure what's going on here with the line art output by /usr/bin/mysql here.
The problem: I'm unable to redirect the line art (used in creating the table columns) to a file!
First, I do this on my terminal.
[sd#host:~/tmp]
$ mysql -usd sd -e 'select * from loan;'
+---------+--------------+--------+
| loan_no | branch_name | amount |
+---------+--------------+--------+
| L-11 | Round Hill | 900 |
| L-14 | Downtown | 1500 |
| L-15 | Perryridge | 1500 |
| L-16 | Perryridge | 1300 |
| L-17 | Downtown | 1000 |
| L-23 | Redwood | 2000 |
| L-93 | Mianus | 500 |
+---------+--------------+--------+
Now, I want this whole blessed thing printed above captured, so I redirect stdout and stderr to the file 'out', like so:
[sd#host:~/tmp]
$ mysql -usd sd -e 'select * from loan;' >out 2>&1
As you can see below, the line art is completely missing!
[sd#host:~/tmp]
$ cat out
loan_no branch_name amount
L-11 Round Hill 900
L-14 Downtown 1500
L-15 Perryridge 1500
L-16 Perryridge 1300
L-17 Downtown 1000
L-23 Redwood 2000
L-93 Mianus 500
More proof that the line art is REALLY missing! (The -T option prints tabs, if any.)
[sd#host:~/tmp]
$ cat -T out
loan_no^Ibranch_name^Iamount
L-11^IRound Hill^I900
L-14^IDowntown^I1500
L-15^IPerryridge^I1500
L-16^IPerryridge^I1300
L-17^IDowntown^I1000
L-23^IRedwood^I2000
L-93^IMianus^I500
So, my question is, how the heck does mysql know who -- whether a terminal or a text file -- is sucking its output from 'its rear end' :-) ?

Program can determine whether output stream is terminal or
not using isatty (3) function.
MySQL uses this information to change its output (see man mysql, "-t" option produces table output for non-interactive mode too)

It knows
The situation is analogous to
$ ls
tom dick harry
and
$ ls > out
$ cat out
tom
dick
harry

Related

MYSQL client issue - Having an access denied issue connecting via Bash Script (WSL2-Ubantu to a localhost windows MySQL DB 8.0 instance

I seem to be having an issue when running a bash script in windows WSL2 / Ubantu, but have no issue if I run the mysql command line with the same params.
This is a fairly straight forward script, albeit , I am new to bash scripting.
#!/bin/bash
PROPERTY_FILE="MySQLDB_glenn.properties"
SCRIPTS_DIR="/home/glenn/scripts/"
echo $SCRIPTS_DIR$PROPERTY_FILE
WSL_HOST_IP=$(ipconfig.exe | awk '/WSL/ {getline; getline; getline; getline; print substr($14, 1, length($14)-1)}')
ALT_WSL_HOST=$WSL_HOST_IP
function atestfunction()
{
myTest=$(echo "This is a test")
echo $myTest
}
function getProperty()
{
PROP_KEY=$1
PROP_VALUE=$(cat $SCRIPTS_DIR$PROPERTY_FILE | grep $PROP_KEY | cut -d'=' -f2)
echo $PROP_VALUE
}
echo "# Reading properties from $PROPERTY_FILE = " $SCRIPTS_DIR$PROPERTY_FILE
DB_USER=$(getProperty "db.username")
DB_PASS=$(getProperty "db.password")
DB_HOST=$(getProperty "db.hostname")
DB_DEFAULT=$(getProperty "db.defaultdb")
echo $DB_USER
echo $DB_PASS
echo $DB_HOST
echo $DB_DEFAULT
echo $ALT_WSL_HOST
echo "Selecting from DB " $DB_DEFAULT
mysql -u$DB_USER -p$DB_PASS -h$ALT_WSL_HOST $DB_DEFAULT -e "use test1; select * from products;"
I setup two versions of the account user , granting it all and with the ability to access via localhost and the second account via 172.0.0.0/255.0.0.0 to handle the fact that WSL comes up with different addresses on reboot.
the variable $ALT_WSL_HOST value is 172.25.208.1 (as I debug it as I type this)
In the Bash script debugger (VS Code) or while running the script ./simpleDBselect.sh, i get
ERROR 1045 (28000): Access denied for user 'wsl_root
'#'172.25.220.8' (using password: YES)
The same mysql command in the same session (i just invoked it in the Terminal Tab of VS Code) or quitting VS code , I get:
mysql -uwsl_root -pxxx-xxxxx -h172.25.208.1 test1 -e "use test1; select * from products;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-----------+----------+-------------+---------------+---------+-------------+-----------+-------------+
| productID | widgetid | productName | productNumber | color | inhouseCost | listPrice | productSize |
+-----------+----------+-------------+---------------+---------+-------------+-----------+-------------+
| 1 | 101 | Widget1 | s1001 | Yellow | 25.00 | 40.00 | medium |
| 2 | 102 | Widget2 | s1002 | Black | 27.00 | 45.00 | small |
| 3 | 103 | Widget3 | s1003 | Black | 28.00 | 40.00 | medium |
| 4 | 104 | Widget4 | s1004 | Red | 21.00 | 34.00 | small |
| 5 | 105 | Widget5 | s1005 | Green | 15.00 | 26.00 | large |
| 6 | 106 | Widget6 | s1006 | Magenta | 40.00 | 75.00 | large |
| 7 | 107 | Widget7 | s1007 | Orange | 50.00 | 85.00 | medium |
| 8 | 108 | Widget8 | s1008 | Blue | 39.00 | 55.00 | small |
| 9 | 109 | Widget9 | s1009 | Gold | 189.00 | 300.00 | large |
+-----------+----------+-------------+---------------+---------+-------------+-----------+-------------+
Does anyone know what I might be missing. I have read loads of documentation and think I have the DB accounts setup correctly for the variable hosts. Note,I have also tried '%' as is the default wildcard . Should I just install MySQL in the ubantu running in WSL as opposed to the windows install. Also note that I am cognizant that WSL running its own virtual ip address , hence why i'm using WSL_HOST_IP=$(ipconfig.exe | awk '/WSL/ {getline; getline; getline; getline; print substr($14, 1, length($14)-1)}').
Any help would be appreciated as I'm at my feeble wits end :)
Best Regards,
Glenn Firester
see above description for my attempts, but wsl_root has DBA and all grants to the tables in db test1

Print column names only once in shell script

Here is my shell script and It is working fine without any errors. But i want to get output differently.
Script:
#!/bin/bash
DB_USER='root'
DB_PASSWD='123456'
DB_NAME='job'
Table_Name='status_table'
#sql=select job_name, date,status from $DB_NAME.$Table_Name where job_name='$f1' and date=CURDATE()
file="/root/jobs.txt"
while read -r f1
do
mysql -N -u$DB_USER -p$DB_PASSWD <<EOF
select job_name, date,status from $DB_NAME.$Table_Name where job_name='$f1' and date=CURDATE()
EOF
done <"$file"
Source Table:
mysql> select * from job.status_table
+---------+----------+------------+-----------+
| Job_id | Job Name | date | status |
+---------+----------+--------+---------------+
| 111 | AA | 2016-12-01 | completed |
| 112 | BB | 2016-12-01 | completed |
| 113 | CC | 2016-12-02 | completed |
| 112 | BB | 2016-12-01 | completed |
| 114 | DD | 2016-12-02 | completed |
| 201 | X | 2016-12-03 | completed |
| 202 | y | 2016-12-04 | completed |
| 203 | z | 2016-12-03 | completed |
| 111 | A | 2016-12-04 | completed |
+---------+----------+------------+-----------+
Input text file
[rteja#server0 ~]# more jobs.txt
AA
BB
CC
DD
X
Y
Z
A
ABC
XYZ
Output - Supressed coumn names
(mysql -N -u$DB_USER -p$DB_PASSWD <<EOF)
[rteja#server0 ~]# ./script.sh
AA 2016-12-01 completed
BB 2016-12-01 completed
Output - without Suppressed column names, output printing the columns names for every loop iteration.
(mysql -u$DB_USER -p$DB_PASSWD <<EOF)
[rteja#server0 ~]# ./script.sh
job_name date status
AA 2016-12-01 completed
job_name date status
BB 2016-12-01 completed
Challenges:
1. Want to print column names only once in output and the result i want to store in CSV file.
2. I don't want to expose password & username in code to everyone. Is there way to hide like i heard we can create environmental variables and call it in the script. And we can set the permissions for the environmental variable file to prevent everyone to access it, and only our should be able to access it.
Rather than executing a select query multiple times you can run a single query as:
job_name in ('AA','BB','CC'...)
To do that first read complete file in an array using mapfile:
mapfile -t arr < jobs.txt
Then format the array values into a list of values suited for IN operator:
printf -v cols "'%s'," "${arr[#]}"
cols="(${cols%,})"
Display your values:
echo "$cols"
('AA','BB','CC','DD','X','Y','Z','A','ABC','XYZ')
Finally run your SQL query as:
mysql -N -u$DB_USER -p$DB_PASSWD <<EOF
select job_name, date,status from $DB_NAME.$Table_Name
where job_name IN "$cols" and date=CURDATE();
EOF
To securely connecting to MySQL use login-paths (.mylogin.cnf)
As per MySQL manual:
The best way to specify server connection information is with your .mylogin.cnf file. Not only is this file encrypted, but any logging of the utility execution does not expose the connection information.

apache/mysql generate too many request on amazon centos

My issue is, my website open too many connections in mysql server this is hanged my website and also generate many requests on apache. i have installed the apache module mod-evasive and mod-security i have also empty the iptables rules basically i am using amazon ec2 so it gives security ip i have blocked all outbound and inbound i have opened only 443,80 and ssh 22 port only but still when i take netstat it shows
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
4 --0.0.0.0
4 ---119.159.195.199
25 ---54.69.254.252
374 ---
on above my question is why my server 374 apache connection on ::1:80 how can block this or reduce this
mysql connection stat are
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Connections | 5208 |
| Threads_cached | 0 |
| Threads_connected | 54 |
| Threads_created | 5207 |
| Threads_running | 54 |
+-------------------+-------+
5 rows in set (0.00 sec)
my second question is why my sql connections is increasing.
Please any one help me i will really appreciated.

Killing sleeping processes in Mysql?

Can anyone tell me how can I kill all the sleeping processes?
I searched for it and I found that we can do it by command
mk-kill --match-command Sleep --kill --victims all --interval 10
I connected the DB server(Linux) but I find the message that command not found.
I tried to connect via MYSQL administrator and it doesn't say that command not found but also doesn't executes the query , just says you have an SQl error
login to Mysql as admin:
mysql -uroot -ppassword;
And than run command:
mysql> show processlist;
You will get something like below :
+----+-------------+--------------------+----------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-------------+--------------------+----------+---------+------+-------+------------------+
| 49 | application | 192.168.44.1:51718 | XXXXXXXX | Sleep | 183 | | NULL ||
| 55 | application | 192.168.44.1:51769 | XXXXXXXX | Sleep | 148 | | NULL |
| 56 | application | 192.168.44.1:51770 | XXXXXXXX | Sleep | 148 | | NULL |
| 57 | application | 192.168.44.1:51771 | XXXXXXXX | Sleep | 148 | | NULL |
| 58 | application | 192.168.44.1:51968 | XXXXXXXX | Sleep | 11 | | NULL |
| 59 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+----+-------------+--------------------+----------+---------+------+-------+------------------+
You will see complete details of different connections. Now you can kill the sleeping connection as below:
mysql> kill 55;
Query OK, 0 rows affected (0.00 sec)
kill $queryID; is helpful but if there is only one query causing an issue;
Having a lot of MySQL sleeping processes can cause a huge spike in your CPU load or IO
Here is a simple one-line command (if behind the MySQL server is linux) which would kill all of the current sleeping MySQL processes:
for i in `mysql -e "show processlist" | awk '/Sleep/ {print $1}'` ; do mysql -e "KILL $i;"; done
This is only a temporary repair; I strongly advise identifying and addressing the problem's main cause.
For instance, you may set the wait timeout variable to the amount of time you want MySQL to hold open connections before shutting them.
But if the issue still persists and you have to investigate the DB queries that cause the problem there is another way. In screen session, you can use another while cycle to continuously kill the sleeping queries. (while there is an output of the mysql show processlit | grep -i sleep | awk id column and kill it.) If you are using MySQL replication between different hosts this will help them to catch up. So when using show slave status\G; Seconds_behind_master will be going to catch up.
Of course, you should investigate the root cause again.

Execute multiline mysql in shellscript

When I attempt to execute a multi-line SQL in mysql via shell script:
mysql -uroot -ppass mydb <<<EOF
SELECT * INTO OUTFILE 'table.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM mytable limit 1;
EOF
I get a syntax error:
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near 'EOF' at line 1
What's the right way to script it?
The syntax for bash heredoc is:
COMMAND <<InputComesFromHERE
...
...
...
InputComesFromHERE
So you have an extra <.
In order to prepare and test you can replace COMMAND (i.e. mysql -uroot -ppass mydb in this case) with cat to have a look at the exact SQL code that will be executed.
For future readers, one easy way is as follows if they wish to export in bulk using bash, tested on mariadb, same should work in mysql too.
akshay#ideapad:/tmp$ mysql -u someuser -p test -e "select * from offices"
Enter password:
+------------+---------------+------------------+--------------------------+--------------+------------+-----------+------------+-----------+
| officeCode | city | phone | addressLine1 | addressLine2 | state | country | postalCode | territory |
+------------+---------------+------------------+--------------------------+--------------+------------+-----------+------------+-----------+
| 1 | San Francisco | +1 650 219 4782 | 100 Market Street | Suite 300 | CA | USA | 94080 | NA |
| 2 | Boston | +1 215 837 0825 | 1550 Court Place | Suite 102 | MA | USA | 02107 | NA |
| 3 | NYC | +1 212 555 3000 | 523 East 53rd Street | apt. 5A | NY | USA | 10022 | NA |
| 4 | Paris | +33 14 723 4404 | 43 Rue Jouffroy D'abbans | NULL | NULL | France | 75017 | EMEA |
| 5 | Tokyo | +81 33 224 5000 | 4-1 Kioicho | NULL | Chiyoda-Ku | Japan | 102-8578 | Japan |
| 6 | Sydney | +61 2 9264 2451 | 5-11 Wentworth Avenue | Floor #2 | NULL | Australia | NSW 2010 | APAC |
| 7 | London | +44 20 7877 2041 | 25 Old Broad Street | Level 7 | NULL | UK | EC2N 1HN | EMEA |
+------------+---------------+------------------+--------------------------+--------------+------------+-----------+------------+-----------+
If you're exporting by non-root user then set permission like below
root#ideapad:/tmp# mysql -u root -p
MariaDB[(none)]> UPDATE mysql.user SET File_priv = 'Y' WHERE user='someuser' AND host='localhost';
Restart or Reload mysqld
akshay#ideapad:/tmp$ sudo su
root#ideapad:/tmp# systemctl restart mariadb
Sample code snippet
akshay#ideapad:/tmp$ cat test.sh
#!/usr/bin/env bash
user="someuser"
password="password"
database="test"
mysql -u"$user" -p"$password" "$database" <<EOF
SELECT *
INTO OUTFILE '/tmp/csvs/offices.csv'
FIELDS TERMINATED BY '|'
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM offices;
EOF
Execute
akshay#ideapad:/tmp$ mkdir -p /tmp/csvs
akshay#ideapad:/tmp$ chmod +x test.sh
akshay#ideapad:/tmp$ ./test.sh
akshay#ideapad:/tmp$ cat /tmp/csvs/offices.csv
"1"|"San Francisco"|"+1 650 219 4782"|"100 Market Street"|"Suite 300"|"CA"|"USA"|"94080"|"NA"
"2"|"Boston"|"+1 215 837 0825"|"1550 Court Place"|"Suite 102"|"MA"|"USA"|"02107"|"NA"
"3"|"NYC"|"+1 212 555 3000"|"523 East 53rd Street"|"apt. 5A"|"NY"|"USA"|"10022"|"NA"
"4"|"Paris"|"+33 14 723 4404"|"43 Rue Jouffroy D'abbans"|\N|\N|"France"|"75017"|"EMEA"
"5"|"Tokyo"|"+81 33 224 5000"|"4-1 Kioicho"|\N|"Chiyoda-Ku"|"Japan"|"102-8578"|"Japan"
"6"|"Sydney"|"+61 2 9264 2451"|"5-11 Wentworth Avenue"|"Floor #2"|\N|"Australia"|"NSW 2010"|"APAC"
"7"|"London"|"+44 20 7877 2041"|"25 Old Broad Street"|"Level 7"|\N|"UK"|"EC2N 1HN"|"EMEA"