I have created a script of hive queries mainly for features creation and scoring for cross sell project. Most of the queries are simple queries that do the data cleaning , transformation etc. I want to automate this process so that I can start with hive table as input and can output the final result into Hbase file . My question are :
What is the best way to do it ?
Can I simply create filename.sql or filename.hql and run it from shell using hive -f filename.sql
Is there something in hive like PL for SQL?
You can do it in multiple ways.
Like you can also use Hive CLI and its very ease to do such jobs.
You can write shell script in Linux or .bat in Windows.
In script you can simply go like below entries.
$HIVE_HOME/bin/hive -e 'select a.col from tab1 a';
or if you have file :
$HIVE_HOME/bin/hive -f /home/my/hive-script.sql
Make sure you have set $HIVE_HOME in your env.
Once you have tested and working fine you can put in cronjob for scheduling.
It is important to note that if you are using either of the technique, each of your queries must be separated by a semi colon i.e.
hive -e 'select * from tableA limit 10;select * from tableB limit 10'
Usually I use manual find to replace text in a MySQL database using phpMyAdmin. I'm tired of it now, how can I run a query to find and replace a text with new text in the entire table in phpMyAdmin?
Example: find keyword domain.example, replace with www.domain.example.
For a single table update
UPDATE `table_name`
SET `field_name` = replace(same_field_name, 'unwanted_text', 'wanted_text')
From multiple tables-
If you want to edit from all tables, best way is to take the dump and then find/replace and upload it back.
The easiest way I have found is to dump the database to a text file, run a sed command to do the replace, and reload the database back into MySQL.
All commands below are bash on Linux.
Dump database to text file
mysqldump -u user -p databasename > ./db.sql
Run sed command to find/replace target string
sed -i 's/oldString/newString/g' ./db.sql
Reload the database into MySQL
mysql -u user -p databasename < ./db.sql
Easy peasy.
Running an SQL query in phpMyAdmin to find and replace text in all WordPress blog posts, such as finding mysite.example/wordpress and replacing that with mysite.example/news
Table in this example is tj_posts
UPDATE `tj_posts`
SET `post_content` = replace(post_content, 'mysite.example/wordpress', 'mysite.example/news')
Put this in a php file and run it and it should do what you want it to do.
// Connect to your MySQL database.
$hostname = "localhost";
$username = "db_username";
$password = "db_password";
$database = "db_name";
mysql_connect($hostname, $username, $password);
// The find and replace strings.
$find = "find_this_text";
$replace = "replace_with_this_text";
$loop = mysql_query("
SELECT
concat('UPDATE ',table_schema,'.',table_name, ' SET ',column_name, '=replace(',column_name,', ''{$find}'', ''{$replace}'');') AS s
FROM
information_schema.columns
WHERE
table_schema = '{$database}'")
or die ('Cant loop through dbfields: ' . mysql_error());
while ($query = mysql_fetch_assoc($loop))
{
mysql_query($query['s']);
}
phpMyAdmin includes a neat find-and-replace tool.
Select the table, then hit Search > Find and replace
This query took about a minute and successfully replaced several thousand instances of oldurl.ext with the newurl.ext within Column post_content
Best thing about this method : You get to check every match before committing.
N.B. I am using phpMyAdmin 4.9.0.1
Another option is to generate the statements for each column in the database:
SELECT CONCAT(
'update ', table_name ,
' set ', column_name, ' = replace(', column_name,', ''www.oldDomain.example'', ''www.newDomain.example'');'
) AS statement
FROM information_schema.columns
WHERE table_schema = 'mySchema' AND table_name LIKE 'yourPrefix_%';
This should generate a list of update statements that you can then execute.
UPDATE table SET field = replace(field, text_needs_to_be_replaced, text_required);
Like for example, if I want to replace all occurrences of John by Mark I will use below,
UPDATE student SET student_name = replace(student_name, 'John', 'Mark');
If you are positive that none of the fields to be updated are serialized, the solutions above will work well.
However, if any of the fields that need updating contain serialized data, an SQL Query or a simple search/replace on a dump file, will break serialization (unless the replaced string has exactly the same number of characters as the searched string).
To be sure, a "serialized" field looks like this:
a:1:{s:13:"administrator";b:1;}
The number of characters in the relevant data is encoded as part of the data.
Serialization is a way to convert "objects" into a format easily stored in a database, or to easily transport object data between different languages.
Here is an explanation of different methods used to serialize object data, and why you might want to do so, and here is a WordPress-centric post: Serialized Data, What Does That Mean And Why is it so Important? in plain language.
It would be amazing if MySQL had some built in tool to handle serialized data automatically, but it does not, and since there are different serialization formats, it would not even make sense for it to do so.
wp-cli
Some of the answers above seemed specific to WordPress databases, which serializes much of its data. WordPress offers a command line tool, wp search-replace, that does handle serialization.
A basic command would be:
wp search-replace 'an-old-string' 'a-new-string' --dry-run
However, WordPress emphasizes that the guid should never be changed, so it recommends skipping that column.
It also suggests that often times you'll want to skip the wp_users table.
Here's what that would look like:
wp search-replace 'https://old-domain.example' 'https://shiney-new-domain.com' --skip-columns=guid --skip-tables=wp_users --dry-run
Note: I added the --dry-run flag so a copy-paste won't automatically ruin anyone's database. After you're sure the script does what you want, run it again without that flag.
Plugins
If you are using WordPress, there are also many free and commercial plugins available that offer a gui interface to do the same, packaged with many additional features.
Interconnect/it PHP script
Interconnect/it offers a PHP script to handle serialized data: Safe Search and Replace tool. It was created for use on WordPress sites, but it looks like it can be used on any database serialized by PHP.
Many companies, including WordPress itself, recommends this tool. Instructions here, about 3/4 down the page.
UPDATE `MySQL_Table`
SET `MySQL_Table_Column` = REPLACE(`MySQL_Table_Column`, 'oldString', 'newString')
WHERE `MySQL_Table_Column` LIKE 'oldString%';
I believe "swapnesh" answer to be the best ! Unfortunately I couldn't execute it in phpMyAdmin (4.5.0.2) who although illogical (and tried several things) it kept saying that a new statement was found and that no delimiter was found…
Thus I came with the following solution that might be usefull if you exeprience the same issue and have no other access to the database than PMA…
UPDATE `wp_posts` AS `toUpdate`,
(SELECT `ID`,REPLACE(`guid`,'http://old.tld','http://new.tld') AS `guid`
FROM `wp_posts` WHERE `guid` LIKE 'http://old.tld%') AS `updated`
SET `toUpdate`.`guid`=`updated`.`guid`
WHERE `toUpdate`.`ID`=`updated`.`ID`;
To test the expected result you may want to use :
SELECT `toUpdate`.`guid` AS `old guid`,`updated`.`guid` AS `new guid`
FROM `wp_posts` AS `toUpdate`,
(SELECT `ID`,REPLACE(`guid`,'http://old.tld','http://new.tld') AS `guid`
FROM `wp_posts` WHERE `guid` LIKE 'http://old.tld%') AS `updated`
WHERE `toUpdate`.`ID`=`updated`.`ID`;
the best you export it as sql file and open it with editor such as visual studio code and find and repalace your words.
i replace in 1 gig file sql in 1 minutes for 16 word that total is 14600 word.
its the best way.
and after replace it save and import it again.
do not forget compress it with zip for import.
In the case of sentences with uppercase - lowercase letters,
We can use BINARY REPACE
UPDATE `table_1` SET `field_1` = BINARY REPLACE(`field_1`, 'find_string', 'replace_string')
Here's an example of how to find and replace in Database
UPDATE TABLE_NAME
SET COLUMN = replace(COLUMN,'domain.example', 'www.domain.example')
TABLE_NAME => Change it with your table name
COLUMN => Change it to your column make sure it exists
I have good luck with this query when doing a search and replace in phpmyadmin:
UPDATE tableName SET fieldName1 = 'foo' WHERE fieldName1 = 'bar';
Of course this only applies to one table at a time.
Generate change SQL queries (FAST)
mysql -e "SELECT CONCAT( 'update ', table_name , ' set ', column_name, ' = replace(', column_name,', ''www.oldsite.example'', ''www.newsite.example'');' ) AS statement FROM information_schema.columns WHERE table_name LIKE 'wp_%'" -u root -p your_db_name_here > upgrade_script.sql
Remove any garbage at the start of the file. I had some.
nano upgrade_script.sql
Run generated script with --force options to skip errors. (SLOW - grab a coffee if big DB)
mysql -u root -p your_db_name_here --force < upgrade_script.sql
I have a c-shell scrip that connects to a mysql database database through and invokes a sql script which in turn invokes another sql script to run a query and return a report
#!/bin/csh
set MYSQL=${MYSQL_HOME}/mysql
set REPORT=${CLEADM_HOME}/Scripts/DataValidation/EOreport.sql
${MYSQL} ${CLEDBUSER} <${REPORT}
Then within the eoreport.sql I invoke another script like so
Source IERSs.sql
and finally in the IERSs.sql script i need to log the results to a log file but it is not working
SELECT *
FROM TB_EARTHORIENTATIONPARAMETER_UI
INTO OUTFILE '/vobs/tools/Scripts /results.log'
This is not working. All i see is the results of the query printed to the xterm(im using tcsh on solaris and the database is mysql client). Am i missing something?
i have even done research about the tee command that is supposed to pipe in you input and output i to the file that you specify as follows
tee /vobs/tools/Scripts/DataValidation/results.txt
SELECT * FROM TB_EARTHORIENTATIONPARAMETER_UI;
but this still outputs results to the screen and leaves my result.txt file empty. What am i missing ?
SELECT *
FROM TB_EARTHORIENTATIONPARAMETER_UI
INTO OUTFILE '/vobs/tools/Scripts /results.log'
you have a extra space between scripts and /, try this:
SELECT *
FROM TB_EARTHORIENTATIONPARAMETER_UI
INTO OUTFILE '/vobs/tools/Scripts/results.log'
Also you said :
"leaves my result.txt file empty." and you are trying to write a result.log file
I have this slot:
void Managment::dbExportTriggered()
{
save = QFileDialog::getSaveFileName(this, trUtf8("Export db"),
QDir::currentPath() + "Backup/",
trUtf8("Dumped database (*.sql)"));
sqlQuery = "SELECT * INTO OUTFILE '" + save + ".sql' FROM Users, Data";
//QMessageBox::critical(0, trUtf8("query dump"), QString::number(query.exec(sqlQuery)));
query.exec(sqlQuery);
}
And I have this query:
sqlQuery = "SELECT * INTO OUTFILE " + save + " FROM Users, Data";
I execute normally but no dumped file appear, the backup directory has the right permission, the dumped database must be in client.
UPDATE:
After a search I found that the INTO OUTFILE query will dump database in the server not in the client as I was thought, so my question now how can I dump database in remote MySQL server, any quick methods with out any external tools like mysqldump client.
SELECT ... INTO OUTFILE creates a file on the MySQL server machine, with permissions matching whoever the MySQL server runs as. Unless you have root access on the MySQL server to retrieve the file that you're exporting, SELECT ... INTO OUTFILE is unlikely to do what you want.
In fact, I think I'd go so far as to say that if you're trying to use SELECT ... INTO OUTFILE from a GUI client, you're probably taking the wrong approach to your problem.
Just an idea: Another approach is to call mysqldump with QProcess. With some google-fu this seems to be an example:
..
if (allDatabases->isChecked()) {
arguments << "--all-databases";
} else {
arguments << "--databases";
foreach(QListWidgetItem *item, databasesList->selectedItems())
arguments << item->text();
}
proc->setReadChannel(QProcess::StandardOutput);
QApplication::setOverrideCursor(Qt::WaitCursor);
proc->start("mysqldump", arguments);
..
Thus, you can also add some parameters to dump only a specific table.
Edit:
Just note from the mysql doc on the SELECT ... INTO OUTFILE statement:
If you want to create the resulting
file on some other host than the
server host, you normally cannot use
SELECT ... INTO OUTFILE since there is
no way to write a path to the file
relative to the server host's file
system.
Thus you must roll your own, or you can use mysql -e as suggested by the above documentation.
Did you dump/print save to check it's valid? Does currentPath() return a trailung "/"?
Could there be difference between the path as seen by your client program and as (to be) seen by the server?
Does the user have the necessary privileges (file privilege for sure, maybe more)
Can't you get an error message from the log?
Are you getting any errors running the sql statement?
I notice that you're concatenating the filename into the SQL query without surrounding it by quotation marks. Your code will yield something like
SELECT * INTO OUTFILE /path/to/somewhere FROM Users, Data
but the MySQL documentation says it wants something like
SELECT * INTO OUTFILE '/path/to/somewhere' FROM Users, Data
Also keep the following in mind:
The file is created on the server host, so you must have the FILE privilege to use this syntax. file_name cannot be an existing file, which among other things prevents files such as /etc/passwd and database tables from being destroyed.
If you're looking on your client, you won't see the file there, even if the operation succeeds.
In PowerShell, how do I execute my mysql script so that the results are piped into a csv file? The results of this script is just a small set of columns that I would like copied into a csv file.
I can have it go directly to the shell by doing:
mysql> source myscript.sql
And I have tried various little things like:
mysql> source myscript.sql > mysql.out
mysql> source myscript.sql > mysql.csv
in infinite variation, and I just get errors. My db connections is alright because I can do basic table queries from the command line etc... I haven't been able to find a solution on the web so far either...
Any help would be really appreciated!
You seem to not be running powershell, but the mysql command line tool (perhaps you started it in a powershell console though.)
Note also that the mysql command line tool cannot export directly to csv.
However, to redirect the output to a file just run
mysql mydb < myscript.sql >mysql.out
or e.g.
echo select * from mytable | mysql mydb >mysql.out
(and whatever arguments to mysql you need, like username, hostname)
Are you looking for SELECT INTO OUTFILE ? dev.mysql.com/doc/refman/5.1/en/select.html – Pekka 19 hours ago
Yep. Select into outfile worked! But to make sure you get column names you also need to do something like:
select *
from
(
select
a,
b,
c
)
Union ALL
(Select *
from actual)