I am trying to query the total count of a partition in BigQuery and store the result in a mysql table. I am doing this for monitoring purpose.
#!/bin/sh
query1="bq query --format=csv 'SELECT COUNT(1) as Count FROM [dataset.tablename] WHERE _PARTITIONTIME = TIMESTAMP(\"$date\")'"
eval result=\$\($query1\)
echo "$result"
bq_insertquery="insert into <<table>>(<<column>>) values(${result})"
echo $bq_insertquery | mysql -u root -p <<dbname>>
Am getting error while insertion in mysql table. This is probably because the variable $result includes both the header and the value, i.e.
Variable $result holds: value with the header
Looks like myquery will be able to insert data, if i get only the value.
How should i assign only value to a shell variable, and use it thereafter ?
Edit: Any sql output contains column_name and values. The variable i assigned to store the value from BigQuery also contains both, i.e. column_name and value. I am looking for something which would be helpfull in extracting only value.
Simply add the --quiet flag (ignore status updates while jobs are running), and pipe it to awk:
query1="bq query --quiet --format=csv 'SELECT COUNT(1) as Count FROM [dataset.tablename] WHERE _PARTITIONTIME = TIMESTAMP(\"$date\")' | awk '{if(NR>1)print}'"
I would like to use jq command to parse the json output from the query. Before that you need to install jq command first. Here is the way to put count into a
result=$(echo -e "select 1 as col" | bq query --nouse_legacy_sql --format=json)
echo $result
it shows the output [{"col":"1"}]. Now it's time to use jq command to get final output.
count=$(echo $count | jq '.[0]' | jq '.col')
echo $count
In One line:
count=$(echo $(echo -e "select 1 as col" | bq query --nouse_legacy_sql --format=json) | jq '.[0]' | jq '.col')
I am trying to get count of column user_Id using count(user_Id) from mysql as follows:
count=$(mysql -uroot -proot csv_imports -e "select count(user_Id) from test_data where user_Id=\"12345\";")
I am not getting what is wrong with it. I want it's numeric result. What could help me?
Using options -B -N in command mysql
--batch, -B - Print results using tab as the column separator, with each row on a new line. With this option, mysql does not use the history file.
--skip-column-names, -N - Do not write column names in results.
count=$(mysql -uroot -proot csv_imports -B -N -e "select count(user_Id) from test_data where user_Id=\"12345\";»)
without options -B -N result is:
+----------------+
| count(user_id) |
+----------------+
| 4 |
+----------------+
with option -B result is:
count(user_id)
4
with option -B -N result is:
4
I sort a query result by using:
ORDER BY STR_TO_DATE(table1.date, '%Y%m%d'); -- the format of `date` is `Ymd` without delimiter.
However, the result is undesired:
20150508
20150514
20150525
20150514
20150515
20150522
20150525
20150529
20150605
20150612
20150619
Sorry for my carefulness. I use the shell command sort (./batch_queries.sh | cut -f 3-9,12- | sort | uniq -c) to sort the query result, leading to such a undsired result.
Is it possible to display query results like below within mysql shell?
mysql> select code, created_at from my_records;
code created_at
1213307927 2013-04-26 09:52:10
8400000000 2013-04-29 23:38:48
8311000001 2013-04-29 23:38:48
3 rows in set (0.00 sec)
instead of
mysql> select code, created_at from my_records;
+------------+---------------------+
| code | created_at |
+------------+---------------------+
| 1213307927 | 2013-04-26 09:52:10 |
| 8400000000 | 2013-04-29 23:38:48 |
| 8311000001 | 2013-04-29 23:38:48 |
+------------+---------------------+
3 rows in set (0.00 sec)
The reason I'm asking because I have some tedious task that I need to copy the output and paste it on other tool.
--raw, -r
For tabular output, the “boxing” around columns enables one column value to be distinguished from another. For nontabular output (such as is produced in batch mode or when the --batch or --silent option is given), special characters are escaped in the output so they can be identified easily. Newline, tab, NUL, and backslash are written as \n, \t, \0, and \\. The --raw option disables this character escaping.
The following example demonstrates tabular versus nontabular output and the use of raw mode to disable escaping:
% mysql
mysql> SELECT CHAR(92);
+----------+
| CHAR(92) |
+----------+
| \ |
+----------+
% mysql --silent
mysql> SELECT CHAR(92);
CHAR(92)
\\
% mysql --silent --raw
mysql> SELECT CHAR(92);
CHAR(92)
\
From MySQL Docs
Not exactly what you need, but it might be useful. Add \G at the end of the query
select code, created_at from my_records\G;
Query result will look like this:
*************************** 1. row ***************************
code: 1213307927
created_at: 2013-04-26 09:52:10
*************************** 2. row ***************************
code: 8400000000
created_at: 2013-04-29 23:38:48
One-liner
mysql -u YOURUSER -p --password=YOURPASSWORD -s -r -e "show databases;"
mysql -u root -p --password=abc12345 -s -r -e "show databases;"
You need to pass the -s parameter mysql -s.
mysql --skip-column-names --silent --raw --execute "select * from somewhere;"
To make these the default options for your user, add the following to ~/.my.cnf
skip_column_names
silent
raw
I solved this but using concat_ws to join the results together and then add a space (first argument)
select concat_ws (' ',ipNetFull,ipUsage,broadcast,gateway) from ipNets;
If you want to get MySQL client output without the surrounding table, you can run your query from the linux command line instead of through the mysql client:
$ echo "SELECT CONCAT_WS(' ','DROP TABLE',TABLE_NAME,';') FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE '%';" | mysql dbname
bonus: you can pipe the output through sort and stuff, or even back into mysql:
$ echo "SELECT CONCAT_WS(' ','DROP TABLE',CONCAT_WS('.',TABLE_SCHEMA,TABLE_NAME),';') FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'i\'m\_not\_that\_mean';" | mysql | mysql;
Quick Workaround:
Using the more recent MySQL clients (not sure which versions), you can use the PAGER directive to manipulate the results. Note that you may have to change the sed expression to avoid clobbering your data.
> PAGER sed 's/|//g'
PAGER set to 'sed 's/|//g''
barracuda#db-ess-rw.us-east-2.ess.aws.vt2.com [ess]
> SELECT 'row1_col1' AS column1, 'row1_col2' AS column2
UNION ALL SELECT 'row2_col1', 'row2_col2';
+---------+---------+
column1 column2
+---------+---------+
row1_col1 row1_col2
row2_col1 row2_col2
+---------+---------+
2 rows in set (0.00 sec)
Beefore you do this, you may want to get the current value of your PAGER:
> PAGER
PAGER set to 'less'
To reset to using 'less' as the PAGER
> PAGER less
PAGER set to 'less'
More fun with PAGER:
https://www.percona.com/blog/2013/01/21/fun-with-the-mysql-pager-command/
So confused by the question and answers. ran sql and result kept displaying without lines or non tabular format. Later realized there is an option when you right click on the query space, navigate to "results to" > "results to grid". This Should set it back to tabular format.
There is also an option for "results to text" which was the format I was experiencing.
:)!
I am using this command:
mysql -u user -ppassword database -e "select distinct entityName,entitySource from AccessControl"
The output is like this:
+-----------------------+--------------+
| entityName | entitySource |
+-----------------------+--------------+
| low | Native |
| high | Native |
| All Groups | AD |
| Help Ser vices Group | AD |
| DEFAULT_USER_GROUP | Native |
| SYSTEM | Native |
| DEFAULT_BA_USER_GROUP | Native |
| soUsersGrp | Native |
+-----------------------+--------------+
My question is: how can I dynamically create an array of variables to store the values entityName and entitySource? What I need to use is use every value of entityName and entitySource to update another table.
Earlier I was trying to store the output in a file and access each line using awk, but that doesn't help because one line may contain multiple words.
Sure, this can be done. I'd like to second the idea that piping mysql to mysql in the shell is awkward, but I understand why it might need to be done (such as when piping mysql to psql or whatever).
mysql -qrsNB -u user -p password database \
-e "select distinct entityName,entitySource from AccessControl" | \
while read record; do
NAME="`echo $record|cut -d' ' -f 1`" # that's a tab delimiter
SOURCE="`echo $record|cut -d' ' -f 2`" # also a tab delimiter
# your command with $NAME and $SOURCE goes here ...
COMMAND="select trousers from namesforpants where entityName='${NAME}'" # ...
echo $COMMAND | mysql # flags ...
done
the -rs flags trim your output down so that you don't have to grok that table thing it gives you, -q asks that the result not be buffered, -B asks for batch mode, and -N asks to not have column names.
What you do with those variables is up to you; probably I would compose statements in that loop and feed those to your subsequent process rather than worry about interpolation and quotes as you have mentioned some of your data has spaces in it. Or you can write/append to a file and then feed that to your subsequent process.
As usual, the manual is your friend. I'll be your friend, too, but the manpage is where the answers are to this stuff. :-)
#!/usr/bin/env bash
mysql -u user -ppassword database -e "select distinct entityName,entitySource from AccessControl" | while read name source; do
echo "entityName: $name, entitySource: $source"
done
Please check it, I fixed it through exec.
[wcuser#localhost]$ temp=`exec mysql -h10.10.8.36 --port=3306 -uwcuser -pwcuser#123 paycentral -e "select endVersion from script_execution_detail where releaseNo='Release1.0' and versionPrefix='PS'"|tail -1`
[wcuser#localhost]$ echo $temp
19