Powershell code for mysql is not working - mysql

I have following code of Powershell where i am trying to sort lastest backup file of mysql database and then try to import this file
I am using the Powershell script for this according to script till the last i get desired o/p and then i copy this o/p and execute in seprate
cmd window it execute smooth but in power shell when i try to do the same thing it fails with following error please help me
Error message
C:\wamp\bin\mysql\mysql5.5.24\bin\mysql.exe --user=root --password=xxx testdest < "C:\mysqltemp\testsrc_2013-12-23_10-46-AM.sql"
cmd.exe : The system cannot find the file specified.
At C:\Users\IBM_ADMIN\AppData\Local\Temp\8a7b4576-97b2-42aa-a0eb-42bb934833a6.ps1:19 char:4
+ cmd <<<< /c " "$pathtomysqldump" --user=$param1 --password=$param2 $param3 < $param5 "
+ CategoryInfo : NotSpecified: (The system cann...file specified.:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
Script is as following
##Select latest file created by Export of mysql dumper
$a=(get-childitem C:\mysqltemp | sort LastWriteTime -Descending | Select-Object Name | select -first 1 -ExpandProperty Name)
$pathtomysqldump = "C:\wamp\bin\mysql\mysql5.5.24\bin\mysql.exe"
#Write-Host "Print variable A -------------"
#$a
$a=$a.Replace(" ", "")
#Write-Host "After Triming ---------------"
#$a
$param1="root"
$param2="xxx"
$param3="testdest"
#$param4="""'<'"""
$param5="""C:\mysqltemp\$a"""
#$p1="$param1 $param2 $param3 < $param5"
# Invoke backup Command. /c forces the system to wait to do the backup
Write-Host " "$pathtomysqldump" --user=$param1 --password=$param2 $param3 < $param5 "
cmd /c " "$pathtomysqldump" --user=$param1 --password=$param2 $param3 < $param5 "
Thanks and Appreciate your help and time for the same.

This is a common misunderstanding involving calling command lines in the Windows operating system, particularly from PowerShell.
I highly recommend using the Start-Process cmdlet to launch a process instead of calling cmd.exe. It's much easier to mentally parse out and understand the path to the executable, and all of the command line parameters separately. The problem with your current script is that you're trying to call an executable file with the following name: C:\wamp\bin\mysql\mysql5.5.24\bin\mysql.exe --user=root --password=xxx testdest < "C:\mysqltemp\testsrc_2013-12-23_10-46-AM.sql", which has been wrapped in a call to cmd.exe. Obviously, that file does not exist, because you're including all of the parameters as part of the filesystem path.
There are too many layers going on here to make it simple to understand. Instead, use Start-Process similar to the following example:
# 1. Define path to mysql.exe
$MySQL = "C:\wamp\bin\mysql\mysql5.5.24\bin\mysql.exe"
# 2. Define some parameters
$Param1 = 'value1';
$Param2 = 'value2';
$Param3 = 'value 3 with spaces';
# 3. Build the command line arguments
# NOTE: Since the value of $Param3 has spaces in it, we must
# surround the value with double quotes in the command line
$ArgumentList = '--user={0} --password={1} "{2}"' -f $Param1, $Param2, $Param3;
Write-Host -Object "Arguments are: $ArgumentList";
# 4. Call Start-Process
# NOTE: The -Wait parameter tells it to "wait"
# The -NoNewWindow parameter prevents a new window from popping up
# for the process.
Start-Process -FilePath $MySQL -ArgumentList $ArgumentList -Wait -NoNewWindow;

Related

Replace a string during the Mysqldump process on Windows

In a powershell script, I have a mysqldump command which outputs to stdin.
The goal is to replace all occurences of a string in that stdin before pushing it into a file, because there is not enough disk space on the machine to hold two separate files (dump is around 30Go).
I have tried this (removed the invoke-expression and mysql args):
mysqldump [...args] | ForEach-Object -Process {$_ -replace 'sourceText','targetText' | Add-Content $dumpDataFile}
Or this:
mysqldump [...args] | Foreach-Object {$_ -replace 'sourceText','targetText'} | Set-Content $dumpDataFile
but it is eating up all the memory on the machine.
I have also tried replacing content in the result file but it always ends up in copying to an another file.
I also thought about reading line by line and replacing line by line to a new file, with each X lines removing lines from the original file, but methods I have found to cut lines in files end up eating all memory.
In linux I would have used sed, I know it exists for windows but I do not want to add a dependency to the script.
Here is the command that is run:
$expr = "& 'C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqldump.exe' --defaults-extra-file=env.cnf --log-error=err.log --no-create-info foo | ForEach-Object -Process {$_ -replace 'foo','bar' | Add-Content dump.sql}"
Invoke-Expression $expr
UPDATE
I have found that even piping out to out-null eats up all the memory:
& 'C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqldump.exe' --defaults-extra-file=env.cnf --log-error=err.log --no-create-info foo | out-null
also the scripts run on an amazon virtual machine which has powershell 4
UPDATE 2
This also eats up all the memory, but it does not when running from cmd:
& 'C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqldump.exe' --defaults-extra-file=env.cnf --log-error=err.log --no-create-info foo > dump.sql
Do you know how to call the full replace command with cmd? I do not manage to escape the mysqldump executable path
UPDATE 3
Realized that my dump contains huge tables, which results in some of the INSERT line being extremely long (thus the memory usage maybe). I tries without extended inserts but it is too long to import then.
If the disk space is premium, how about compressing the data? If NTFS compression isn't good enough, let's write the output into a GZipStream. It should offer good savings for text data. Thus the file on disk would be considerably smaller.
First off, a compression function (idea from a blog post):
function Compress-Stream {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true, ValueFromPipeline=$true)]
[AllowEmptyString()]
[string]$Row
)
begin {
$ms = New-Object System.IO.MemoryStream
$cs = New-Object System.IO.Compression.GZipStream($ms, [System.IO.Compression.CompressionMode]::Compress)
$sw = New-Object System.IO.StreamWriter($cs)
}
process {
if(-not [string]::IsNullOrWhiteSpace($row)) {
$sw.Write($Row + [environment]::NewLine)
}
}
end {
try {$cs.Close(); $cs.Dispose()} catch{}
try {$sw.Close(); $sw.Dispose()} catch{}
$s = [System.Convert]::ToBase64String($ms.ToArray());
try {$ms.Close(); $ms.Dispose()} catch {}
$s
}
}
Sample usage is to query DBA Overflow data dump. Tt's much more manageable that SO. On my system the result set is 13 MB uncompressed, 3,5 MB compressed.
# SQL Server, so sqlcmd for illustration.
# Pipe results to compression and pipe compressed data into a file
sqlcmd -E -S .\sqli001 -d dbaoverflow -Q "select id, postid from votes order by id;" `
| compress-stream | Set-Content -Encoding ascii -Path c:\temp\data.b64
This should provide a compressed text file. To process it, use MemoryStream and GZipStream again:
$d = get-content c:\temp\data.b64
$data = [System.Convert]::FromBase64String($d)
$ms = New-Object System.IO.MemoryStream
$ms.Write($data, 0, $data.Length)
$ms.Seek(0,0) | Out-Null
$sr = New-Object System.IO.StreamReader(New-Object System.IO.Compression.GZipStream($ms, [System.IO.Compression.CompressionMode]::Decompress))
# $sr can now read decompressed data. For example,
$sr.ReadLine()
id postid
$sr.ReadLine()
----------- -----------
$sr.ReadLine()
1 2
Doing replacements and writing the final result into another a file should be easy enough.
In the end I use python to replace the string in the dump file while sending it to mysql.
It is fast enough and low on memory.

Shell script function with global variable

yesterday I got a very easy task, but unfortunatelly looks like i can't do with a nice code.
The task briefly: I have a lot of parameters, that I want to ask with whiptail "interactive" mode in the installer script.
The detail of code:
#!/bin/bash
address="192.168.0.1" # default address, what the user can modify
addressT="" # temporary variable, that I want to check and modify, thats why I don't modify in the function the original variable
port="1234"
portT=""
... #there is a lot of other variable that I need for the installer
function parameter_set {
$1=$(whiptail --title "Installer" --inputbox "$2" 12 60 "$3" 3>&1 1>&2 2>&3) # thats the line 38
}
parameter_set "addressT" "Please enter IP address!" "$address"
parameter_set "portT" "Please enter PORT!" "$port"
But i got the following error:
"./install.sh: line: 38: address=127.0.0.1: command not found"
If I modify the variable to another (not a parameter of function), works well.
function parameter_set {
foobar=$(whiptail --title "Installer" --inputbox "$2" 12 60 "$3" 3>&1 1>&2 2>&3)
echo $foobar
}
I try to use global retval variable, and assign to outside of the function to the original variable, it works, but I think it's not the nicest solution for this task.
Could anybody help me, what I do it wrong? :)
Thanks in advance (and sorry for my bad english..),
Attila
It seems that your whiptail command is not producing anyoutput because of the redirections. So the command substitution leaves the value empty. Try removing them. Also it's better to save the new value to a local variable first:
parameter_set() {
local NAME=$1
local NEWVALUE=$(whiptail --title "Installer" --inputbox "$2" 12 60 "$3")
export $NAME="$NEWVALUE"
}

Newbie: unix bash, nested if statement, results from a loop results from sql

Newbie here, please pardon any confusing wording that I use.
A common task I have is to take a list of names and do a MySQL query to look the names up in a table and see if they are "live" on our site.
Doing this one at a time, my SQL query works fine. I then wanted to do the query using a loop from a file listing multiple names. This works fine, too.
I added this query loop to my bash profile so that I can quickly do the task by typing this:
$ ValidOnSite fileName
This works fine, and I even added an usage statement for my process to remind myself of the syntax. Below is what I have that works fine:
validOnSite() {
if [[ "$1" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then
echo "Usage:"
echo " $ validOnSite [filename]"
echo " Where validOnSite uses specified file as variables in sql query:"
echo " SELECT name, active FROM dbDb WHERE name=lines in file"
else
cat $1 | while read line ; do hgsql -h genome-centdb hgcentral -Ne "select name, active from dbDb where name='$line'" ; done
fi
Using a file "list.txt" which contains:
nameA
nameB
I would then type:
validOnSite list.txt
and both entries in list.txt meet my query criteria and are found in sql. My results will be:
nameA 1
nameB 1
Note the "1" after each result. I assume this is some sort of "yes" status.
Now, I add a third name to my list.txt, one that I know is not a match in sql. Now list.txt contains:
nameA
nameB
foo
When I again run this command for my list with 3 rows:
validOnSite list.txt
My results are the same as when I used the 1st version of file.txt, and I cannot see which lines failed, I still only see which lines were a success:
nameA 1
nameB 1
I have been trying all kinds of things to add a nested if statement, something that says, "If $line is a match, echo "pass", else echo "fail."
I do not want to see a "1" in my results. Using file.txt with 2 matches and 1 non-match, I would like my results to be:
nameA pass
nameB pass
foo fail
Or even better, color code a pass with green and a fail with red.
As I said, newbie here... :)
Any pointers in the right direction would help. Here is my latest sad attempt, but I realize I may be going in a wrong direction entirely:
validOnSite() {
if [[ "$1" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then
echo "Usage:"
echo " $ validOnSite [filename]"
echo " Where validOnSite uses specified file as variables in sql query:"
echo " SELECT name, active FROM dbDb WHERE name=lines in file"
else
cat $1 | while read line ; do hgsql -h genome-centdb hgcentral -Ne "select name, active from dbDb where name='$line'" > /dev/null ; done
if ( "status") then
echo $line "failed"
echo $line "failed" >> outfile
else
echo $line "ok"
echo $line "ok" >>outfile
clear
cat outfile
fi
fi
If something looks crazy in my last attempt, it's because it is - I am just googling around and trying as many things as I can while trying to learn. Any help appreciated, I feel stuck after working on this for a long time, but I am excited to move forward and find a solution! I think there is something I'm missing about understanding stdout, and also confusion about nested if's.
Note: I do not need an outfile, but it's ok if one is needed to accomplish the goal. stdout result alone would suffice, and is preferred.
Note: hgssql is just the name of our MySQL server. The MySQL part works fine, I am looking for a better way to deal with my bash output, and I think there is something about stderr that I'm missing. I'm looking for a fairly simple answer as I'm a newbie!
I guess, by hgsql you mean some Mercurial extension that allows to perform MySQL queries. I don't know how hgsql works, but I know that MySQL returns only the matching rows. But in terms of shell scripting, the result is a string that may contain extra information even if the number of matched rows is zero. For example, some MySQL client may return the header or a string like "No rows found", although it is unlikely.
I'll show how it is done with the official mysql client. I'm sure you will manage to adapt hgsql with the help of its documentation to the following example.
if [ -t 1 ]; then
red_color=$(tput setaf 1)
green_color=$(tput setaf 2)
reset_color=$(tput sgr0)
else
red_color=
green_color=
reset_color=
fi
colorize_flag() {
local color
if [ "$1" = 'fail' ]; then
color="$red_color"
else
color="$green_color"
fi
printf '%s' "${color}${1}${reset_color}"
}
sql_fmt='SELECT IF(active, "pass", "fail") AS flag FROM dbDb WHERE name = "%s"'
while IFS= read -r line; do
sql=$(printf "$sql_fmt" "$line")
flag=$(mysql --skip-column-names dbname -e "$sql")
[ -z "$flag" ] && flag='fail'
printf '%-20s%s\n' "$line" "$(colorize_flag "$flag")"
done < file
The first block detects if the script is running in interactive mode by checking if the file descriptor 1 (standard output) is opened on a terminal (see help test). If it is opened in a terminal, the script considers that the script is running interactively, i.e. the standard output is connected to the user's terminal directly, but not via pipe, for example. For interactive mode, it assigns variables to the terminal color codes with the help of tput command.
colorize_flag function accepts a string ($1) and outputs the string with the color codes applied according to its value.
The last block reads file line by line. For each line builds an SQL query string (sql) and invokes mysql command with the column names stripped off the output. The output of the mysql command is assigned to flag by means of command substitution. If "$flag" is empty, it is assigned to 'fail'. The $line and the colorized flag are printed to standard output.
You can test the non-interactive mode by chaining the output via pipe, e.g.:
./script | tee -a
I must warn you that it is generally bad idea to pass the shell variables into SQL queries unless the values are properly escaped. And the popular shells do not provide any tools to escape MySQL strings. So consider running the queries in Perl, PHP, or any programming language that is capable of building and running the queries safely.
Also note that in terms of performance it is better to run a single query and then parse the result set in a loop instead of running multiple queries in a loop, with the exception of prepared statements.
I found a way to get to my solution by piecing together the few basic things that I know. Not elegant, but it works well enough for now. I created a file "[filename]Results" with the output:
nameA 1
nameB 1
I then cut out the "1"s and made a new file. I then did a comparison with "[fileName]results" to list.txt in order to see what lines exist in file.txt but do not exist in results.
Note: I have the following in my .zshrc file.
validOnSite() {
if [[ "$1" == "" ]] || [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then
echo "Usage:"
echo " $ validOnSite [filename]"
echo " Where validOnSite uses specified file as variables in sql query:"
echo " SELECT name, active FROM dbDb WHERE name=lines in file"
else
cat $1 | while read line ; do hgsql -h genome-centdb hgcentral -Ne "select name from dbDb where name='$line' and active='1'" >> $1"Pass"; done
autoload -U colors
colors
echo $fg_bold[magenta]Assemblies active on site${reset_color}
echo
cat $1"Pass"
echo
echo $fg_bold[red]Not active or not found on site${reset_color}
comm -23 $1 $1"Pass" 2> /dev/null
echo
echo
mv $1"Pass" ~cath/myFiles/validOnSiteResults
echo "Results file containing only active assemblies resides in ~cath/myFiles/validOnSiteResults"
fi
}
list.txt:
nameA
nameB
foo
My input:
validOnSite list.txt
My output:
Assemblies active on site (<--this font is magenta)
nameA
nameB
Not active or not found on site (<--this font is red)
foo
Results file containing only active assemblies resides in ~me/myFiles/validOnRRresults

PowerShell MySQL Backup Script Error in Task Scheduler 0x00041301

I have created the following PowerShell script.
$root = 'C:\Backups\My Website\Database Dumps\'
$dateString = (Get-Date).ToString("yyyy-MM-dd")
$fileName = $dateString + "-MyWebsiteDbBackup.sql"
$backupFilePath = ($root + $fileName)
$command = ("mysqldump -u root wpdatabase > " + "`"$backupFilePath`"")
Write-Host $command
Invoke-Expression $command
Its function is supposed to be making a daily backup of a MySQL database for my WordPress website.
When I run the script in PowerShell ISE, it runs fine and the MySQL dump file is created with no problems.
However, in Task Scheduler, it was stuck on running with a code 0x00041301.
For the credentials, I am using the my.cnf technique described here. And I've set the task to run whether a user is logged on or not.
CODE UPDATE
Based on vonPryz's answer.
$root = 'C:\Backups\My Website\Database Dumps\'
$dateString = (Get-Date).ToString("yyyy-MM-dd")
$fileName = $dateString + "-MyWebsiteDbBackup.sql"
$backupFilePath = ($root + $fileName + " 2>&1")
$command = ("mysqldump -u root wpdatabase > " + "`"$backupFilePath`"")
Write-Host $command
$output = Invoke-Expression $command
$output | Out-File C:\mysqlBackupScriptOutput.txt
This now give me an error saying illegal character in path
What am I doing wrong?
Task Scheduler's code 0x00041301 means that the task is running. This is likely to mean that mysqldump is prompting for something. Maybe a password or some confirmation dialog. Which user account is the task being run on?
In order to debug, you'd need to capture the process' output to see what's going on. Try using Tee-Object to send a copy to a file.

Can i access 'dxdiag ' from powershell console

How can I access "dxdiag" with powershell . I want to run a script for gather some information from a few remote computers.
If you use the /x parameter, you can have dxdiag output to an xml file, which is then really easily parsed from powershell. Basically just something like this:
# Drop output in temp dir
$logFile = $env:TEMP + "\dxDiagLog.xml"
# Piping to Out-Null forces it to wait for dxdiag to complete before continuing. Otherwise
# it tries to load the file before it actuallygets written
dxdiag.exe /whql:off /dontskip /x $logFile | Out-Null
[xml]$dxDiagLog = Get-Content $logFile
$dxDiagLog.DxDiag.DirectSound.SoundDevices.SoundDevice | ft Description, DriverProvider
Which dumps this for output on my machine:
Description DriverProvider
----------- --------------
Speakers (Realtek High Definition Audio) Realtek Semiconductor Corp.
Polycom CX700 (Polycom CX700) Microsoft
In my case the command would run, and only later would it create the file.
(& dxdiag.exe /whql:off /dontskip /t `"$path`") | Out-Null
The problem is with the ampersand & which made the command exit before completion.
So either use:
dxdiag.exe /whql:off /dontskip /x $logFile | Out-Null
Or:
Start-Process -FilePath "C:\Windows\System32\dxdiag.exe" -ArgumentList "/dontskip /whql:off /t C:\Temp\dxdiag.txt" -Wait
From: https://community.spiceworks.com/topic/2116806-how-can-i-run-dxdiag-on-a-remote-pc