Execute multiline mysql in shellscript - mysql

When I attempt to execute a multi-line SQL in mysql via shell script:
mysql -uroot -ppass mydb <<<EOF
SELECT * INTO OUTFILE 'table.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM mytable limit 1;
EOF
I get a syntax error:
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near 'EOF' at line 1
What's the right way to script it?

The syntax for bash heredoc is:
COMMAND <<InputComesFromHERE
...
...
...
InputComesFromHERE
So you have an extra <.
In order to prepare and test you can replace COMMAND (i.e. mysql -uroot -ppass mydb in this case) with cat to have a look at the exact SQL code that will be executed.

For future readers, one easy way is as follows if they wish to export in bulk using bash, tested on mariadb, same should work in mysql too.
akshay#ideapad:/tmp$ mysql -u someuser -p test -e "select * from offices"
Enter password:
+------------+---------------+------------------+--------------------------+--------------+------------+-----------+------------+-----------+
| officeCode | city | phone | addressLine1 | addressLine2 | state | country | postalCode | territory |
+------------+---------------+------------------+--------------------------+--------------+------------+-----------+------------+-----------+
| 1 | San Francisco | +1 650 219 4782 | 100 Market Street | Suite 300 | CA | USA | 94080 | NA |
| 2 | Boston | +1 215 837 0825 | 1550 Court Place | Suite 102 | MA | USA | 02107 | NA |
| 3 | NYC | +1 212 555 3000 | 523 East 53rd Street | apt. 5A | NY | USA | 10022 | NA |
| 4 | Paris | +33 14 723 4404 | 43 Rue Jouffroy D'abbans | NULL | NULL | France | 75017 | EMEA |
| 5 | Tokyo | +81 33 224 5000 | 4-1 Kioicho | NULL | Chiyoda-Ku | Japan | 102-8578 | Japan |
| 6 | Sydney | +61 2 9264 2451 | 5-11 Wentworth Avenue | Floor #2 | NULL | Australia | NSW 2010 | APAC |
| 7 | London | +44 20 7877 2041 | 25 Old Broad Street | Level 7 | NULL | UK | EC2N 1HN | EMEA |
+------------+---------------+------------------+--------------------------+--------------+------------+-----------+------------+-----------+
If you're exporting by non-root user then set permission like below
root#ideapad:/tmp# mysql -u root -p
MariaDB[(none)]> UPDATE mysql.user SET File_priv = 'Y' WHERE user='someuser' AND host='localhost';
Restart or Reload mysqld
akshay#ideapad:/tmp$ sudo su
root#ideapad:/tmp# systemctl restart mariadb
Sample code snippet
akshay#ideapad:/tmp$ cat test.sh
#!/usr/bin/env bash
user="someuser"
password="password"
database="test"
mysql -u"$user" -p"$password" "$database" <<EOF
SELECT *
INTO OUTFILE '/tmp/csvs/offices.csv'
FIELDS TERMINATED BY '|'
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM offices;
EOF
Execute
akshay#ideapad:/tmp$ mkdir -p /tmp/csvs
akshay#ideapad:/tmp$ chmod +x test.sh
akshay#ideapad:/tmp$ ./test.sh
akshay#ideapad:/tmp$ cat /tmp/csvs/offices.csv
"1"|"San Francisco"|"+1 650 219 4782"|"100 Market Street"|"Suite 300"|"CA"|"USA"|"94080"|"NA"
"2"|"Boston"|"+1 215 837 0825"|"1550 Court Place"|"Suite 102"|"MA"|"USA"|"02107"|"NA"
"3"|"NYC"|"+1 212 555 3000"|"523 East 53rd Street"|"apt. 5A"|"NY"|"USA"|"10022"|"NA"
"4"|"Paris"|"+33 14 723 4404"|"43 Rue Jouffroy D'abbans"|\N|\N|"France"|"75017"|"EMEA"
"5"|"Tokyo"|"+81 33 224 5000"|"4-1 Kioicho"|\N|"Chiyoda-Ku"|"Japan"|"102-8578"|"Japan"
"6"|"Sydney"|"+61 2 9264 2451"|"5-11 Wentworth Avenue"|"Floor #2"|\N|"Australia"|"NSW 2010"|"APAC"
"7"|"London"|"+44 20 7877 2041"|"25 Old Broad Street"|"Level 7"|\N|"UK"|"EC2N 1HN"|"EMEA"

Related

MYSQL client issue - Having an access denied issue connecting via Bash Script (WSL2-Ubantu to a localhost windows MySQL DB 8.0 instance

I seem to be having an issue when running a bash script in windows WSL2 / Ubantu, but have no issue if I run the mysql command line with the same params.
This is a fairly straight forward script, albeit , I am new to bash scripting.
#!/bin/bash
PROPERTY_FILE="MySQLDB_glenn.properties"
SCRIPTS_DIR="/home/glenn/scripts/"
echo $SCRIPTS_DIR$PROPERTY_FILE
WSL_HOST_IP=$(ipconfig.exe | awk '/WSL/ {getline; getline; getline; getline; print substr($14, 1, length($14)-1)}')
ALT_WSL_HOST=$WSL_HOST_IP
function atestfunction()
{
myTest=$(echo "This is a test")
echo $myTest
}
function getProperty()
{
PROP_KEY=$1
PROP_VALUE=$(cat $SCRIPTS_DIR$PROPERTY_FILE | grep $PROP_KEY | cut -d'=' -f2)
echo $PROP_VALUE
}
echo "# Reading properties from $PROPERTY_FILE = " $SCRIPTS_DIR$PROPERTY_FILE
DB_USER=$(getProperty "db.username")
DB_PASS=$(getProperty "db.password")
DB_HOST=$(getProperty "db.hostname")
DB_DEFAULT=$(getProperty "db.defaultdb")
echo $DB_USER
echo $DB_PASS
echo $DB_HOST
echo $DB_DEFAULT
echo $ALT_WSL_HOST
echo "Selecting from DB " $DB_DEFAULT
mysql -u$DB_USER -p$DB_PASS -h$ALT_WSL_HOST $DB_DEFAULT -e "use test1; select * from products;"
I setup two versions of the account user , granting it all and with the ability to access via localhost and the second account via 172.0.0.0/255.0.0.0 to handle the fact that WSL comes up with different addresses on reboot.
the variable $ALT_WSL_HOST value is 172.25.208.1 (as I debug it as I type this)
In the Bash script debugger (VS Code) or while running the script ./simpleDBselect.sh, i get
ERROR 1045 (28000): Access denied for user 'wsl_root
'#'172.25.220.8' (using password: YES)
The same mysql command in the same session (i just invoked it in the Terminal Tab of VS Code) or quitting VS code , I get:
mysql -uwsl_root -pxxx-xxxxx -h172.25.208.1 test1 -e "use test1; select * from products;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-----------+----------+-------------+---------------+---------+-------------+-----------+-------------+
| productID | widgetid | productName | productNumber | color | inhouseCost | listPrice | productSize |
+-----------+----------+-------------+---------------+---------+-------------+-----------+-------------+
| 1 | 101 | Widget1 | s1001 | Yellow | 25.00 | 40.00 | medium |
| 2 | 102 | Widget2 | s1002 | Black | 27.00 | 45.00 | small |
| 3 | 103 | Widget3 | s1003 | Black | 28.00 | 40.00 | medium |
| 4 | 104 | Widget4 | s1004 | Red | 21.00 | 34.00 | small |
| 5 | 105 | Widget5 | s1005 | Green | 15.00 | 26.00 | large |
| 6 | 106 | Widget6 | s1006 | Magenta | 40.00 | 75.00 | large |
| 7 | 107 | Widget7 | s1007 | Orange | 50.00 | 85.00 | medium |
| 8 | 108 | Widget8 | s1008 | Blue | 39.00 | 55.00 | small |
| 9 | 109 | Widget9 | s1009 | Gold | 189.00 | 300.00 | large |
+-----------+----------+-------------+---------------+---------+-------------+-----------+-------------+
Does anyone know what I might be missing. I have read loads of documentation and think I have the DB accounts setup correctly for the variable hosts. Note,I have also tried '%' as is the default wildcard . Should I just install MySQL in the ubantu running in WSL as opposed to the windows install. Also note that I am cognizant that WSL running its own virtual ip address , hence why i'm using WSL_HOST_IP=$(ipconfig.exe | awk '/WSL/ {getline; getline; getline; getline; print substr($14, 1, length($14)-1)}').
Any help would be appreciated as I'm at my feeble wits end :)
Best Regards,
Glenn Firester
see above description for my attempts, but wsl_root has DBA and all grants to the tables in db test1

Import excel data put into the SQL database?

I am tried to import excel data into my MySQL Query browser database. I try to use below the coding type in the MySQL Query to execute, but it cannot work.
My code:
LOAD DATA LOCAL INFILE
'c:/2019/countries.csv'
INTO TABLE countries
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(id,name,country_code,language);
The My SQL query browser show me the error:
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ROWS
(id,name,country_code,language)' at line 7
My database info:
My excel data info (My excel file location is"c:/2019/countries.csv"):
My CSV file info:
id,name,country_code,language
231,Andorra,20,en
232,United Arab Emirates,784,en
233,Afghanistan,4,en
234,Antigua and Barbuda,28,en
235,Anguilla,660,en
236,Albania,8,en
237,Armenia,51,en
238,Angola,24,en
239,Antarctica,10,en
240,Argentina,32,en
241,American Samoa,16,en
Anyone can guide me what I get the wrong in my coding? Thanks.
Depending on your MySQL version, IGNORE clauses in LOAD DATA exclusively uses LINES or in latter versions, interchangebaly uses LINES or ROWS. You can see this by docs of different versions:
LINES and ROWS are synonymous
[IGNORE number {LINES | ROWS}]
MySQL 8
MySQL 5.7: Load Data
MySQL 5.6: Load Data
MySQL 5.5: Load Data
Only LINES allowed (i.e., ROWS not supported)
[IGNORE number LINES]
MySQL 5.4: Load Data
MySQL 5.1: Load Data
Not a problem (win10)
drop table if exists t;
create table t
(id int,name varchar(20),country_code int,language varchar(20));
LOAD DATA LOCAL INFILE
'C:\\Program Files\\MariaDB 10.1\\data\\sandbox\\data.txt'
INTO TABLE t
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 ROWS
(id,name,country_code,language);
+------+----------------------+--------------+----------+
| id | name | country_code | language |
+------+----------------------+--------------+----------+
| 231 | Andorra | 20 | en |
| 232 | United Arab Emirates | 784 | en |
| 233 | Afghanistan | 4 | en |
| 234 | Antigua and Barbuda | 28 | en |
| 235 | Anguilla | 660 | en |
| 236 | Albania | 8 | en |
| 237 | Armenia | 51 | en |
| 238 | Angola | 24 | en |
| 239 | Antarctica | 10 | en |
| 240 | Argentina | 32 | en |
| 241 | American Samoa | 16 | en |
+------+----------------------+--------------+----------+
11 rows in set (0.00 sec)

Sqoop incremental import and update does not work

How can I update the data in HDFS file similar to data in MySQL table?
I checked the internet, but all the examples given are with --incremental lastmodified example.
Where in my case my MySQL table does not contain a date or timestamp column.
How can I update the data in HDFS file similar to data in MySQL table that does not contain date column?
I have MySQL table as below
mysql> select * from employee;
+----+--------+--------+------+-------+-----------+
| id | name | gender | age | state | language |
+----+--------+--------+------+-------+-----------+
| 1 | user1 | m | 25 | tn | tamil |
| 2 | user2 | m | 41 | ka | tamil |
| 3 | user3 | f | 47 | kl | tamil |
| 4 | user4 | f | 52 | ap | telugu |
| 5 | user5 | m | 55 | ap | telugu |
| 6 | user6 | f | 43 | tn | tamil |
| 7 | user7 | m | 34 | tn | malayalam |
| 8 | user8 | f | 33 | ap | telugu |
| 9 | user9 | m | 36 | ap | telugu |
I imported to HDFS using the below command.
[cloudera#localhost ~]$ sqoop import --connect jdbc:mysql://localhost:3306/mydatabase --username root --table employee --as-textfile --target-dir hdfs://localhost.localdomain:8020/user/cloudera/data/employee
The data is imported as expected.
[cloudera#localhost ~]$ hadoop fs -ls /user/cloudera/data/employee/
Found 6 items
-rw-r--r-- 3 cloudera cloudera 0 2017-08-16 23:57 /user/cloudera/data/employee/_SUCCESS
drwxr-xr-x - cloudera cloudera 0 2017-08-16 23:56 /user/cloudera/data/employee/_logs
-rw-r--r-- 3 cloudera cloudera 112 2017-08-16 23:56 /user/cloudera/data/employee/part-m-00000
-rw-r--r-- 3 cloudera cloudera 118 2017-08-16 23:56 /user/cloudera/data/employee/part-m-00001
-rw-r--r-- 3 cloudera cloudera 132 2017-08-16 23:56 /user/cloudera/data/employee/part-m-00002
-rw-r--r-- 3 cloudera cloudera 136 2017-08-16 23:56 /user/cloudera/data/employee/part-m-00003
Now I updated values and inserted values in mysql table. But this table doesnot contain date column.
mysql> update employee set language = 'marathi' where id >= 8;
mysql> insert into employee (name,gender,age,state,language from people) values('user11','f','25','kl','malayalam');
I know the newly inserted values can be inserted to hdfs using --check-column, incremental append and --last-value.
But how can I update the values in hdfs for the mysql table rows 8 and 9 that were updated to 'marathi'? Also, my employee table does not contain a date or timestamp column.
For newly inserted row, you can always use:
--incremental append --check-column id --last-value 9
But for getting updates from table not having updated_at column, I don't think thats possible. If your table is very small, then probably just do a full dump every time.
Or if you somehow can maintain track of what all ids got updated since last import, then let's say you know ids 7, 3, 4 and 8 got updated since last import, you can use the minimum of updated ids and use as --last-value. So your config will be:
--incremental append --check-column id --last-value 3 --merge-key id
where --merge-key id will tell sqoop to merge the new incremental data with old based on id column.

MySql Query regarding use of TOP command

ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near '(2) from empdata where empid='E-713'' at line 1
mysql> delete TOP(2) from empdata where empid='E-713';
+-------+--------+--------+--------+------+---------+
| id | name | height | salary | age | city |
+-------+--------+--------+--------+------+---------+
| E-713 | Rajat | 5.11 | 25000 | 25 | jaipur |
| E-720 | Ritesh | 5.8 | 30000 | 27 | Delhi |
| E-711 | Javed | 5.7 | 23000 | 25 | kashmir |
| E-715 | Puneet | 5.1 | 20000 | 27 | Noida |
| E-713 | Rajat | 5.11 | 25000 | 25 | jaipur |
+-------+--------+--------+--------+------+---------+
How can I get nth highest salary using top command in MySql. although my syntax is correct as per my knowledge but on pressing enter the above error flashed.
This is a bit long for a comment.
MySQL does not support SELECT TOP. That is usually associated with SQL Server.
It does support LIMIT, so you could write:
delete ed
from empdata ed
where empid = 'E-713'
limit 2;
However, this is very dangerous, because it deletes two arbitrary rows. In almost all cases, you want an ORDER BY:
delete ed
from empdata ed
where empid = 'E-713'
order by ??
limit 2;
This is true whether you are using TOP or LIMIT.

Unable to understand this strange MySql client behavior

I'm not sure what's going on here with the line art output by /usr/bin/mysql here.
The problem: I'm unable to redirect the line art (used in creating the table columns) to a file!
First, I do this on my terminal.
[sd#host:~/tmp]
$ mysql -usd sd -e 'select * from loan;'
+---------+--------------+--------+
| loan_no | branch_name | amount |
+---------+--------------+--------+
| L-11 | Round Hill | 900 |
| L-14 | Downtown | 1500 |
| L-15 | Perryridge | 1500 |
| L-16 | Perryridge | 1300 |
| L-17 | Downtown | 1000 |
| L-23 | Redwood | 2000 |
| L-93 | Mianus | 500 |
+---------+--------------+--------+
Now, I want this whole blessed thing printed above captured, so I redirect stdout and stderr to the file 'out', like so:
[sd#host:~/tmp]
$ mysql -usd sd -e 'select * from loan;' >out 2>&1
As you can see below, the line art is completely missing!
[sd#host:~/tmp]
$ cat out
loan_no branch_name amount
L-11 Round Hill 900
L-14 Downtown 1500
L-15 Perryridge 1500
L-16 Perryridge 1300
L-17 Downtown 1000
L-23 Redwood 2000
L-93 Mianus 500
More proof that the line art is REALLY missing! (The -T option prints tabs, if any.)
[sd#host:~/tmp]
$ cat -T out
loan_no^Ibranch_name^Iamount
L-11^IRound Hill^I900
L-14^IDowntown^I1500
L-15^IPerryridge^I1500
L-16^IPerryridge^I1300
L-17^IDowntown^I1000
L-23^IRedwood^I2000
L-93^IMianus^I500
So, my question is, how the heck does mysql know who -- whether a terminal or a text file -- is sucking its output from 'its rear end' :-) ?
Program can determine whether output stream is terminal or
not using isatty (3) function.
MySQL uses this information to change its output (see man mysql, "-t" option produces table output for non-interactive mode too)
It knows
The situation is analogous to
$ ls
tom dick harry
and
$ ls > out
$ cat out
tom
dick
harry