How to use ADODB in PowerShell - mysql

I thought I'd post these code snippets for others who may find themselves trying to make ADODB calls from a PowerShell script. I inherited this convoluted mess, and had to make some changes to it.
We're using PlanetPress as part of a Docuware Document Imaging system. A PP workflow called a vbscript which in turn launched a PowerShell script. The PowerShell did the work to make two database queries. One was an update, and the other was a select. I'm not that great with PowerShell, and there may be a cmdlet out there to simplify this. But the code was creating ADODB.Connection, ADODB.Command, and ADODB.Resultset objects directly. The problem is there are no good resources for the syntax required to use these objects. Hopefully these code snippets will help some poor soul in a similar situation.
Using ADODB.Command:
$oConnection = New-Object -comobject "ADODB.Connection"
# Use correct ODBC driver
if ([Environment]::Is64BitProcess) {
$oConnection.Open("DSN=DW64")
} else {
$oConnection.Open("DSN=DW")
}
if ($oConnection.State -eq $adStateClosed) {
Write-Output "Connection not established"
Write-Output $oConnection
}
$UpdQuery = "Update dwdata.Purchasing `
set Status='Processing' `
WHERE DOCUMENT_TYPE = 'Check' `
AND STATUS in ('Approved')"
$ra=-1
$oCommand = New-Object -comobject "ADODB.Command"
$oCommand.ActiveConnection = $oConnection
$oCommand.CommandText = $UpdQuery
$oCommand.CommandType = $adCmdText
$rs=$oCommand.Execute([ref]$ra)
Write-Output ("Count of Row[s] updated: " + $ra)
Using ADODB.Resultset:
$oRS = New-Object -comobject "ADODB.Recordset"
$query = "SELECT DWDOCID, DOCUMENT_DATE, CHECK_NUMBER, PAYEE_NAME, CHECK_AMOUNT, STATUS `
FROM dwdata.Purchasing `
WHERE DOCUMENT_TYPE = 'Check' `
AND STATUS = 'Processing' `
ORDER BY CHECK_NUMBER;"
# $oConnection object created in ADODB.Command snippet above
$oConnection.CursorLocation = $adUseClient
$oRS.Open($query, $oConnection, $adOpenStatic, $adLockOptimistic)
$reccount = "Number of queried records: " + $oRS.RecordCount
write-output $reccount
If (-not ($oRS.EOF)) {
# Move to the first record returned, and loop
$oRS.MoveFirst()
$reccount = "Number of loop records: " + $oRS.RecordCount
write-output $reccount
do {
$outString = '"' + $oRS.Fields.Item("DOCUMENT_DATE").Value.ToString("MM/dd/yyyy") + '"' + ','
$outString += '"' + $oRS.Fields.Item("CHECK_NUMBER").Value + '"' + ','
$outString += '"' + $oRS.Fields.Item("PAYEE_NAME").Value + '"' + ','
$outString += '"' + $oRS.Fields.Item("CHECK_AMOUNT").Value + '"' + ','
$outString | Out-File $bankfile -Append -Encoding ASCII
$oRS.MoveNext()
} until
($oRS.EOF -eq $True)
} Else{
Write-Output "No records returned from database query."
}
$oRS.Close()
$oConnection.Close()
Some of this code is ugly (using do instead of while), but the idea is to help you get the right syntax for $oCommand.Execute and how to get a record count from the Recordset. $oRS.MoveFirst() needs to be called before the record count is available.
ss64.com and other resources usually give vbscript snippets. In vbscript variables are not preceeded with a $, and when or if you need to use parenthesis is unclear. This code does run and work.

Related

Why my PowerShell script not run as expected

I have created a script to crawl the IMDB website. My script take a list of IMDB urls, run and extract the data like movie title, release year, plot summary and export it to a text file in CSV. I wrote the script as below.
$listToCrawl = "imdb_link_list.txt"
$pathOfFile = "K:\MY DOCUMENTS\POWERSHELL\IMDB FILE\"
$fileName = "plot_summary.txt"
New-Item ($pathOfFile + $fileName) -ItemType File
Set-Content ($pathOfFile + $fileName) '"Title","Year","URL","Plot Summary"'
Get-Content ($pathOfFile + $listToCrawl) | ForEach-Object {
$url = $_
$Result = Invoke-WebRequest -Uri $url
$movieTitleSelector = "#title-overview-widget > div.vital > div.title_block > div > div.titleBar > div.title_wrapper > h1"
$movieTitleNode = $Result.ParsedHtml.querySelector( $movieTitleSelector)
$movieTitle = $movieTitleNode.innerText
$movieYearSelector = "#titleYear"
$movieYearNode = $Result.ParsedHtml.querySelector($movieYearSelector)
$movieYear = $movieYearNode.innerText
$plotSummarySelector = "#titleStoryLine > div:nth-child(3) > p > span"
$plotSummaryNode = $Result.ParsedHtml.querySelector($plotSummarySelector)
$plotSummary = $plotSummary.innerText
$movieDataEntry = '"' + $movieTitle + '","' + $movieYear + '","' + $url + '","' + $plotSummary + '"'
Add-Content ($pathOfFile + $fileName) $movieDataEntry
}
The list of urls to extract from is saved in the "K:\MY DOCUMENTS\POWERSHELL\IMDB FILE\imdb_link_list.txt" file and the content is as below.
https://www.imdb.com/title/tt0472033/
https://www.imdb.com/title/tt0478087/
https://www.imdb.com/title/tt0285331/
https://www.imdb.com/title/tt0453562/
https://www.imdb.com/title/tt0120577/
https://www.imdb.com/title/tt0416449/
I just import and run the script. It does not run as expected. The error is threw.
Invalid argument.
At K:\MY DOCUMENTS\POWERSHELL\IMDB_Plot_Summar_ Extract.ps1:20 char:1
+ $plotSummaryNode = $Result.ParsedHtml.querySelector($plotSummarySelec ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException
I think the problem is due to the CSS selector I use to select the data but I don't know what's wrong. I think I have followed the CSS selector rule.
$plotSummarySelector = "#titleStoryLine > div:nth-child(3) > p > span"
Does anyone know what's wrong with the thing.
The ParsedHtml property is specific to PowerShell for Windows and doesn't exist in PowerShell Core, so if you want to future-proof your code you're better off using something like the HTML Agility Pack.
# install the HTML Agility Pack nuget package
Invoke-WebRequest -Uri "https://dist.nuget.org/win-x86-commandline/latest/nuget.exe" -OutFile ".\nuget.exe";
.\nuget.exe install "HtmlAgilityPack" -Version "1.11.21";
# import the HTML Agility Pack
Add-Type -Path ".\HtmlAgilityPack.1.11.21\lib\Net40\HtmlAgilityPack.dll";
# get the web page content and load it into a HtmlDocument
$response = Invoke-WebRequest -Uri "https://www.imdb.com/title/tt0472033/" -UseBasicParsing;
$html = $response.Content;
$doc = new-object HtmlAgilityPack.HtmlDocument;
$doc.LoadHtml($html);
then you can extract nodes using XPath syntax - e.g. for the title:
# extract the title
$titleHtml = $doc.DocumentNode.SelectSingleNode("//div[#class='title_wrapper']/h1/text()[1]").InnerText;
$titleText = [System.Net.WebUtility]::HtmlDecode($titleHtml).Trim();
write-host "'$titleText'"; # '9'
I'll leave the rest of the document elements as an exercise for the reader :-).

Powershell Export-CSV from MySQL database reader fails mid-export

I'm a bit new to PowerShell, and I've got a new requirement to get Data out of a MySQL database and into an Oracle one. The strategy I chose was to output to a CSV and then import the CSV into Oracle.
I wanted to get a progress bar for the export from MySQL into CSV, so I used the data reader to achieve this. It works, and begins to export, but somewhere during the export (around record 5,000 of 4.5mil -- not consistent) it will throw an error:
Exception calling "Read" with "0" argument(s): "Fatal error encountered during data read." Exception calling "Close" with "0" argument(s): "Timeout in IO operation" Method invocation failed because [System.Management.Automation.PSObject] does not contain a method named 'op_Addition'. Exception calling "ExecuteReader" with "0" argument(s): "The CommandText property has not been properly initialized."
Applicable code block is below. I'm not sure what I'm doing wrong here, and would appreciate any feedback possible. I've been pulling my hair out on this for days.
Notes: $tableObj is a custom object with a few string fields to hold table name and SQL values. Not showing those SQL queries here, but they work.
Write-Host "[INFO]: Gathering data from MySQL select statement..."
$conn = New-Object MySql.Data.MySqlClient.MySqlConnection
$conn.ConnectionString = $MySQLConnectionString
$conn.Open()
#
# Get Count of records in table
#
$countCmd = New-Object MySql.Data.MySqlClient.MySqlCommand($tableObj.SqlCount, $conn)
$recordCount = 0
try{
$recordCount = $countCmd.ExecuteScalar()
} Catch {
Write-Host "[ERROR]: (" $tableObj.Table ") Error getting Count."
Write-Host "---" $_.Exception.Message
Exit
}
$recordCountString = $recordCount.ToString('N0')
Write-Host "[INFO]: Count for table '" $tableObj.Table "' is " $recordCountString
#
# Compose the command
#
$cmd = New-Object MySql.Data.MySqlClient.MySqlCommand($tableObj.SqlExportInit, $conn)
#
# Write to CSV using DataReader
#
Write-Host "[INFO]: Data gathered into memory. Writing data to CSV file '" $tableObj.OutFile "'"
$counter = 0 # Tracks items selected
$reader=$cmd.ExecuteReader()
$dataRows = #()
# Read all rows into a hash table
while ($reader.Read())
{
$counter++
$percent = ($counter/$recordCount)*100
$percentString = [math]::Round($percent,3)
$counterString = $counter.ToString('N0')
Write-Progress -Activity '[INFO]: CSV Export In Progress' -Status "$percentString% Complete" -CurrentOperation "($($counterString) of $($recordCountString))" -PercentComplete $percent
$row = #{}
for ($i = 0; $i -lt $reader.FieldCount; $i++)
{
$row[$reader.GetName($i)] = $reader.GetValue($i)
}
# Convert hashtable into an array of PSObjects
$dataRows += New-Object psobject -Property $row
}
$conn.Close()
$dataRows | Export-Csv $tableObj.OutFile -NoTypeInformation
EDIT: Didn't work, but I also added this line to my connection string: defaultcommandtimeout=600;connectiontimeout=25 per MySQL timeout in powershell
Using #Carl Ardiente's thinking, the query is timing out, and you have to set the timeout to something insane to fully execute. You simply have to set the timeout value for your session before you start getting data.
Write-Host "[INFO]: Gathering data from MySQL select statement..."
$conn = New-Object MySql.Data.MySqlClient.MySqlConnection
$conn.ConnectionString = $MySQLConnectionString
$conn.Open()
# Set timeout on MySql
$cmd = New-Object MySql.Data.MySqlClient.MySqlCommand("set net_write_timeout=99999; set net_read_timeout=99999", $conn)
$cmd.ExecuteNonQuery()
#
# Get Count of records in table
#
...Etc....
Not that I've found the solution, but none of the connection string changes worked. Manually setting the timeout didn't seem to help either. It seemed to be caused from too many rows returned, so I broke up the function to run in batches, and append to a CSV as it goes. This gets rid of the IO / timeout error.

Get Value of JSON Sting in PowerShell

First of all thank you in advance. I am trying to retrieve the following values from the JSON string below using PowerShell but am having issues.
ScheduleTitle,
Title,
Environment and Title
Resolution,
Width,
Height
JSON:
{"Id":"d52fb00e-8736-448c-a496-96db6bc2eb43","ScheduleId":"275726dc-09f2-4869-b1a0-54c71ef6a093","ScheduleTitle":"Resolution Testing","RunType":"RunNow","Timestamp":"2017-08-01T04:52:19.2039685","AutomationRunItems":[{"Id":"f7b731fb-cd96-4003-b95c-ef1594f1c86e","AutomationRunId":"d52fb00e-8736-448c-a496-96db6bc2eb43","Status":"Done","Case":{"Id":"c8a0b939-54ba-4b98-8ca9-101093aec26f","Title":"Resolution Test"},"Environment":{"Id":"e1783fb5-4001-45d0-9971-b49c31b374ab","Title":"Se Chrome"},"Keyframes":[{"Timestamp":"2017-08-01T04:52:03.0685609","Elapsed":"00:00:00","Level":"Info","Status":"Connecting","BlockId":"78c198ac-b61f-40eb-b1d9-e706546f2be3","LogMessage":"Connecting to Se Chrome: Selenium Grid (localhost:5559) on Chrome with size 1280x1024"},{"Timestamp":"2017-08-01T04:52:03.0685609","Elapsed":"00:00:00.0000003","Level":"Info","Status":"Connected","BlockId":"78c198ac-b61f-40eb-b1d9-e706546f2be3","LogMessage":"Connected"},{"Timestamp":"2017-08-01T04:52:03.0685609","Elapsed":"00:00:00.0000015","Level":"Info","Status":"Running","BlockId":"78c198ac-b61f-40eb-b1d9-e706546f2be3","LogMessage":"Running"},{"Timestamp":"2017-08-01T04:52:03.0695642","Elapsed":"00:00:00.0003729","Level":"Trace","Status":"Running","BlockId":"78c198ac-b61f-40eb-b1d9-e706546f2be3","LogMessage":"Block is executing."},{"Timestamp":"2017-08-01T04:52:03.5713896","Elapsed":"00:00:00.5020154","Level":"Trace","Status":"Running","BlockId":"78c198ac-b61f-40eb-b1d9-e706546f2be3","LogMessage":"Block is executed."},{"Timestamp":"2017-08-01T04:52:03.5713896","Elapsed":"00:00:00.5021370","Level":"Trace","Status":"Running","BlockId":"302f21b8-a279-48ce-9338-580a97c930fd","LogMessage":"Block is executing."},{"Timestamp":"2017-08-01T04:52:06.9793063","Elapsed":"00:00:03.9107459","Level":"Info","Status":"Running","BlockId":"302f21b8-a279-48ce-9338-580a97c930fd","LogMessage":"Chrome was successfully started"},{"Timestamp":"2017-08-01T04:52:09.4641639","Elapsed":"00:00:06.3955903","Level":"Info","Status":"Running","BlockId":"302f21b8-a279-48ce-9338-580a97c930fd","LogMessage":"Chrome loaded url https://dev.bamapplication.com/app/starke/npr/3BF6DA09C1/wizard"},{"Timestamp":"2017-08-01T04:52:09.7604882","Elapsed":"00:00:06.6925088","Level":"Trace","Status":"Running","BlockId":"302f21b8-a279-48ce-9338-580a97c930fd","LogMessage":"Block is executed."},{"Timestamp":"2017-08-01T04:52:09.7604882","Elapsed":"00:00:06.6926471","Level":"Trace","Status":"Running","BlockId":"26382d5d-60a6-4343-b934-265c4e97186d","LogMessage":"Block is executing."},{"Timestamp":"2017-08-01T04:52:10.4639126","Elapsed":"00:00:07.3954580","Level":"Warning","Status":"Running","BlockId":"26382d5d-60a6-4343-b934-265c4e97186d","LogMessage":"Web screenshot is saved"},{"Timestamp":"2017-08-01T04:52:10.7666916","Elapsed":"00:00:07.6982198","Level":"Trace","Status":"Running","BlockId":"26382d5d-60a6-4343-b934-265c4e97186d","LogMessage":"Block is executed."},{"Timestamp":"2017-08-01T04:52:10.7666916","Elapsed":"00:00:07.6983727","Level":"Trace","Status":"Running","BlockId":"161e621f-bb7a-4f42-a5f1-67e07d1432f4","LogMessage":"Block is executing."},{"Timestamp":"2017-08-01T04:52:11.7677637","Elapsed":"00:00:08.6992111","Level":"Info","Status":"Done","BlockId":"161e621f-bb7a-4f42-a5f1-67e07d1432f4","LogMessage":"Case is stopped."}],"Resolution":{"Width":1280,"Height":1024},"Elapsed":"00:00:08.6992111","CreatedAt":"2017-08-01T04:52:19.2039685","ModifiedAt":"2017-08-01T04:52:19.2039685"}],"ProjectId":"275726dc-09f2-4869-b1a0-54c71ef6a093","ExecutionTotalTime":"00:00:08.6992111","FailedCount":0,"PassedCount":0,"DoneCount":1,"CreatedAt":"2017-08-01T04:52:03.0685609"}
Code:
Param( [string]$result, [string]$rootPath )
try{
$json = $result
$parsed = $json | ConvertFrom-Json
$output= ''
foreach ($line in $parsed | Get-Member) {
Write-Output $parsed.$($line.Resolution).property1
Write-Output $parsed.$($line.Resolution).property2
$output += $parsed.$($line.Resolution).property1 + " " + $parsed.$($line.Resolution).property1
}
}
With your $parsed data you can extract all the information you need.
For example to extract the width:
echo $parsed.AutomationRunItems.Resolution.Width
should print: 1280
I think these are all the fields you need extracted. Note, since you did not specify the output precisely I have simply used comma-delimited strings:
$output= ''
$output = $output + $parsed.ScheduleTitle + ','
$output = $output + $parsed.AutomationRunItems.Case.Title + ','
$output = $output + $parsed.AutomationRunItems.Environment.Title + ','
$output = $output + $parsed.AutomationRunItems.Resolution.Width + ','
$output = $output + $parsed.AutomationRunItems.Resolution.Height
Should Print: Resolution Testing,Resolution Test,Se Chrome,1280,1024

Powershell 2.0 - loop through records in CSV for FTP Login validate and update csv with confirmation

Current issue, probably stupid simple but have been grinding my wheels for awhile. I have 600+ FTP accounts to validate if they are able to be logged on. Verified field to be updated in CSV.
CSV Sample:
Connection,Name,Path,Password,Phonetic,Client Name,Client ID,Verified
FTP,BFGftp,d:\data\ftpaccounts\BFGftp,8Wu8Agec8$,(Eight - WHISKEY - uniform - Eight - ALPHA - golf - echo - charlie - Eight - Dollar),(Nine - papa - ECHO - foxtrot - ROMEO - Two - kilo - echo - NOVEMBER - Two),,
FTP,bookitftp,d:\data\ftpaccounts\flipitftp,i3439flip12##,,Flipitftp,30342,
Missing statement block after 'else' keyword.
At C:\Users\mgoeres\Desktop\B-SWIFT FTP TEST2.ps1:118 char:10
+ else <<<< ($i++){ $GetFTP }
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : MissingStatementBlockAfterElse
FTP ACCOUNT VALIDATOR
Function Get-FtpDirectory($Directory) {
#Example
#$Server = "ftp://ftp.example.com/"
#$User = "anonymous#example.com"
#$Pass = "anonymous#anonymous.com"
$Server = "ftp://ftp.localhost.com/"
$User = $FTPLogon
$Pass = $FTPPass
# Credentials
$FTPRequest = [System.Net.FtpWebRequest]::Create("$($Server)$($Directory)")
$FTPRequest.Credentials = New-Object System.Net.NetworkCredential($User,$Pass)
$FTPRequest.Method = [System.Net.WebRequestMethods+FTP]::ListDirectoryDetails
# Don't want Binary, Keep Alive unecessary.
$FTPRequest.UseBinary = $False
$FTPRequest.KeepAlive = $False
$FTPResponse = $FTPRequest.GetResponse()
$ResponseStream = $FTPResponse.GetResponseStream()
# Create a nice Array of the detailed directory listing
$StreamReader = New-Object System.IO.Streamreader $ResponseStream
$DirListing = (($StreamReader.ReadToEnd()) -split [Environment]::NewLine)
$StreamReader.Close()
# Remove first two elements ( . and .. ) and last element (\n)
$DirListing = $DirListing[2..($DirListing.Length-2)]
# Close the FTP connection so only one is open at a time
$FTPResponse.Close()
# This array will hold the final result
$FileTree = #()
# Loop through the listings
foreach ($CurLine in $DirListing) {
# Split line into space separated array
$LineTok = ($CurLine -split '\ +')
# Get the filename (can even contain spaces)
$CurFile = $LineTok[8..($LineTok.Length-1)]
# Figure out if it's a directory. Super hax.
$DirBool = $LineTok[0].StartsWith("d")
# Determine what to do next (file or dir?)
If ($DirBool) {
# Recursively traverse sub-directories
$FileTree += ,(Get-FtpDirectory "$($Directory)$($CurFile)/")
}
Else {
# Add the output to the file tree
$FileTree += ,"$($Directory)$($CurFile)"
}
}
if (!$FileTree -eq $null) {
$Verified = "Y"
Return $FileTree
}
else {
$Verified = "Failed"}
}
# Update data record field "Verified" in CSV '$data' file with status value[Y,Failed,Other]
There is where the import of the FTP login ID and Password should start
#Column Names [Connection,Name,Path,Password,Phonetic,Client Name,Client ID,Verified]
#Variables
$AllName = #()
$AllPassword = #()
$AllVerified = #()
$data = #()
$FTPLogon = #()
$FTPPass = #()
$response = #()
$GetFTP = Get-FtpDirectory
# Import CSV File "FTP_Account_List_with_Password_and_Path.csv"
$data = Import-Csv C:\Users\mgoeres\Desktop\FTP_Account_List_with_Password_and_Path.csv | where-object {$_.Connection -eq "FTP"}
# This import allows key columns to be referenced as variables.
#Import-Csv C:\Users\mgoeres\Desktop\FTP_Account_List_with_Password_and_Path.csv | ForEach-Object {
#$AllName += $_.Name
#$AllPassword += $_.Password
#$AllVerified += $_.Verified}
# Simple test to see if CSV was imported into '$data' and column headers are working as variables
$data|select-object Name,Connection,Verified
Write-Host "CSV File Opened and Loaded"
## Select (next)record from '$data' and store correlating values in '$UserName' and '$Password'
# .EXAMPLE '$data[0].Password' <= First records Password value.
# .EXAMPLE '$data[1].Name' <= Second records Name value.
for ($i=0; $i -lt $data.count; $i++){
$FTPLogon = $data[$i].Name
$FTPPass = $data[$i].Password
$Verified = $data[$i].Verified }
if ($Verifed -eq $null){ $GetFTP }
else ($i++){ $GetFTP }

Join fields in excel using powershell

(1..$numrows) | ForEach-Object {
$sheet.Cells.Item($_,1) = -join $sheet.Cells.Item($numrows,1) + '-1234';
}
I am trying to join the -1234 in a row of a csv file
My result is System.__ComObject-1234
Could you please advise about the error message?
1 - You cannot start an expression with a switch, -join is not a command. Example use : $myJoinedArray = $array -join ";"
2 - You don't need -join, + is enough to concatenate strings.
3 - I think you want $sheet.Cells.Item($numrows,1).Value :
(1..$numrows) | ForEach-Object {
$sheet.Cells.Item($_,1) = $sheet.Cells.Item($numrows,1).Value + '-1234';
}