Error When Calculating Standard Deviation in PowerShell - csv

I'm attempting to calculate some statistics for the values in each column of a CSV file using PowerShell. The Measure-Object cmdlet seems like it will do the trick for everything I need other than the Standard Deviation. I tracked down a description online where the Standard Deviation is calculated using [MATH], but when I run the code, I get the following error on the lines containing Pow():
Method invocation failed because system.management.automation.psobject doesn't contain a method named 'op_Subtraction'.
Here's my code, any help would be appreciated:
$i = 1
While ($i -le 211) {
#Set the variable to the filename with the iteration number
$filename = "c:\zMFM\z550Output\20dSummer\fixed20dSum550Output$i.csv"
#Check to see if that a file with $filename exists. If not, skip to the next iteration of $i. If so, run the code to collect the statistics for each variable and output them each to a different file
If (Test-Path $filename) {
#Calculate the Standard Deviation
#First get the average of the values in the column
$STDEVInputFile = Import-CSV $filename
#Find the average and count for column 'td'
$STDEVAVG = $STDEVInputFile | Measure-Object td -Average | Select Count, Average
$DevMath = 0
# Sum the squares of the differences between the mean and each value in the array
Foreach ($Y in $STDEVInputFile) {
$DevMath += [math]::pow(($Y - $STDEVAVG.Average), 2)
#Divide by the number of samples minus one
$STDEV = [Math]::sqrt($DevMath / ($STDEVAVG.Count-1))
}
#Calculate the basic statistics for column 'td' with the MEASURE-OBJECT cmdlet
$STATS = Import-CSV $Filename |
Measure-Object td -ave -max -min
#Append the standard deviation variable to the statistics array and add the value
$colSTDDEV = New-Object System.Data.DataColumn StdDev,([double])
$colVZA = New-Object System.Data.DataColumn VZA,([double])
$colVAZ = New-Object System.Data.DataColumn VAZ,([double])
$VZA = $Stats.VZA
$VAZ = $Stats.VAZ
$STATS.Columns.Add($colSTDDEV)
$STATS[0].StandardDev = $STDEV
$STATS.Columns.Add($colVZA)
$STATS[0].StandardDev = $VZA
$STATS.Columns.Add($colVAZ)
$STATS[0].StandardDev = $VAZ
#Export the $STATS file containing everything you need in the correct folder
Export-CSV -notype "c:\zMFM\z550Output\20dSummer\20dSum550Statistics.csv"
}
$i++
}

$DevMath += [math]::pow(($Y - $STDEVAVG.Average), 2)
You may have to replace this with:
$DevMath += [math]::pow(($Y.Average - $STDEVAVG.Average), 2)
since $Y seems to be an object, not a numeric value.

Related

Can't get it to provide proper output

I have this function and I can not get it to work, the $DecimalConversion output is not coming out.. I think I am having some syntax errors.
function Get-DecimalNumber(){
$FileCheck = Test-Path "C:\Conversions\conversions.csv"
if($FileCheck){
Do
{
[int]$GetDecimal = Read-host "Write a number between 1-255" | Out-Null
}
while ($GetDecimal -notmatch '\d{1,3}' -or (1..255) -notcontains $GetDecimal)
$DecimalConversion= "{0:X}" -f $GetDecimal
$DecimalConversion
}
else{Write-Warning "Can not find conversions.csv, creating now under C:\Conversions\"; New-Item "C:\Conversions\conversions.csv" -Force | Out-Null}
}
$getfunction=Get-DecimalNumber
You could probably use a better while condition. However, ur issue is caused because of the out-null cmdlet on read-host.
If you use that, $GetDecimal will not get the value you pass in since the out-null is processed before the assignment happens. Just remove it. And it should work.
Final code, I think this looks better, let me know what you think!
function Get-DecimalNumber {
<#
.Description
The Get-DecimalNumber function gets user input for a decimal number and
converts it into hexadecimal and binary numbers. Then this data is added to an
excel file (.csv) and the date of conversion is displayed in short form m/d/yyyy.
#>
$ErrorActionPreference = 'silentlycontinue' #Silences errors
$Test = Test-Path "C:\temp\test\conversions.csv" #Variable to test path
if (! $Test) { #Checking if path does not exist
Write-Warning "conversions.csv File Not Present, creating under C:\temp\test\"
New-Item 'C:\temp\test\conversions.csv' -Force | Out-Null; break; exit #Creating path with file & suppressing output
}
else {
[int]$Num = Read-Host "Enter number from 1-255"
if ($Num -gt 255 -or $Num -le 1) {
Write-Warning "You did not enter a number in the specified range"; break; exit
}
else {
$Hex = [Convert]::ToString($Num, 16) #Converting from decimal to hexadecimal
$Bin = [Convert]::ToString($Num, 2) #Converting from decimal to binary
Write-Host "Decimal to Hex and Binary:"
$NewHashTable1 = #{ } #Creating hashtable
$NewHashTable1.Add('Decimal', $Num) #Adding values from variables to hash table
$NewHashTable1.Add('Hexadecimal', $hex)
$NewHashTable1.Add('Binary', $bin)
$NewHashTable1 #Output to screen the previously created hashtable
$NewHashTable1 >> "C:\temp\test\conversions.csv" #Appending hashtable to .csv file
Write-Output "'n"
Get-Date -Format d #Output date in short format
$Now = Get-Date -Format d
$Now >> "C:\temp\test\conversions.csv" #Output date to .csv file
}
}
}

How to clone a PowerShell PSCustomObject variable to disconnect it from another variable? [duplicate]

I have a powershell script in which I do the following
$somePSObjectHashtables = New-Object Hashtable[] $somePSObject.Length;
$somePSObjects = Import-CSV $csvPath
0..($somePSObject.Length - 1) | ForEach-Object {
$i = $_;
$somePSObjectHashtables[$i] = #{};
$somePSObject[$_].PSObject.Properties | ForEach-Object {
$somePSObjectHashtables[$i][$_.Name] = $_.Value;
}
}
I need to do this because I want to make several distinct copies of the data in the CSV to perform several distinct manipulations. In a sense I'm performing an "INNER JOIN" on the resulting array of PSObject. I can easily iterate through $somePSObjectHashtables with a ForEach-Object and call Hashtable.Clone() on each member of the array. I can then use New-Object PSObject -Property $someHashTable[$i] to get a deep copy of the PSObject.
My question is, is there some easier way of making the deep copy, without an intermediary Hashtable?
Note that here is a shorter, maybe a bit cleaner version of this (that I quite enjoy):
$data = Import-Csv .\test.csv
$serialData = [System.Management.Automation.PSSerializer]::Serialize($data)
$data2 = [System.Management.Automation.PSSerializer]::Deserialize($serialData)
Note:
However, weirdly, it does not keep the ordering of ordered hashtables.
$data = [ordered] #{
1 = 1
2 = 2
}
$serialData = [System.Management.Automation.PSSerializer]::Serialize($data)
$data2 = [System.Management.Automation.PSSerializer]::Deserialize($serialData)
$data2
Will output:
Name Value
---- -----
2 2
1 1
While with other types it works just fine:
$data = [PsCustomObject] #{
1 = 1
2 = 2
}
$data = #(1, 2, 3)
For getting really deep copies we can use binary serialization (assuming that all data are serializable; this is definitely the case for data that come from CSV):
# Get original data
$data = Import-Csv ...
# Serialize and Deserialize data using BinaryFormatter
$ms = New-Object System.IO.MemoryStream
$bf = New-Object System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
$bf.Serialize($ms, $data)
$ms.Position = 0
$data2 = $bf.Deserialize($ms)
$ms.Close()
# Use deep copied data
$data2
Here's an even shorter one that I use as a function:
using namespace System.Management.Automation
function Clone-Object ($InputObject) {
<#
.SYNOPSIS
Use the serializer to create an independent copy of an object, useful when using an object as a template
#>
[psserializer]::Deserialize(
[psserializer]::Serialize(
$InputObject
)
)
}

Powershell 2.0 - loop through records in CSV for FTP Login validate and update csv with confirmation

Current issue, probably stupid simple but have been grinding my wheels for awhile. I have 600+ FTP accounts to validate if they are able to be logged on. Verified field to be updated in CSV.
CSV Sample:
Connection,Name,Path,Password,Phonetic,Client Name,Client ID,Verified
FTP,BFGftp,d:\data\ftpaccounts\BFGftp,8Wu8Agec8$,(Eight - WHISKEY - uniform - Eight - ALPHA - golf - echo - charlie - Eight - Dollar),(Nine - papa - ECHO - foxtrot - ROMEO - Two - kilo - echo - NOVEMBER - Two),,
FTP,bookitftp,d:\data\ftpaccounts\flipitftp,i3439flip12##,,Flipitftp,30342,
Missing statement block after 'else' keyword.
At C:\Users\mgoeres\Desktop\B-SWIFT FTP TEST2.ps1:118 char:10
+ else <<<< ($i++){ $GetFTP }
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : MissingStatementBlockAfterElse
FTP ACCOUNT VALIDATOR
Function Get-FtpDirectory($Directory) {
#Example
#$Server = "ftp://ftp.example.com/"
#$User = "anonymous#example.com"
#$Pass = "anonymous#anonymous.com"
$Server = "ftp://ftp.localhost.com/"
$User = $FTPLogon
$Pass = $FTPPass
# Credentials
$FTPRequest = [System.Net.FtpWebRequest]::Create("$($Server)$($Directory)")
$FTPRequest.Credentials = New-Object System.Net.NetworkCredential($User,$Pass)
$FTPRequest.Method = [System.Net.WebRequestMethods+FTP]::ListDirectoryDetails
# Don't want Binary, Keep Alive unecessary.
$FTPRequest.UseBinary = $False
$FTPRequest.KeepAlive = $False
$FTPResponse = $FTPRequest.GetResponse()
$ResponseStream = $FTPResponse.GetResponseStream()
# Create a nice Array of the detailed directory listing
$StreamReader = New-Object System.IO.Streamreader $ResponseStream
$DirListing = (($StreamReader.ReadToEnd()) -split [Environment]::NewLine)
$StreamReader.Close()
# Remove first two elements ( . and .. ) and last element (\n)
$DirListing = $DirListing[2..($DirListing.Length-2)]
# Close the FTP connection so only one is open at a time
$FTPResponse.Close()
# This array will hold the final result
$FileTree = #()
# Loop through the listings
foreach ($CurLine in $DirListing) {
# Split line into space separated array
$LineTok = ($CurLine -split '\ +')
# Get the filename (can even contain spaces)
$CurFile = $LineTok[8..($LineTok.Length-1)]
# Figure out if it's a directory. Super hax.
$DirBool = $LineTok[0].StartsWith("d")
# Determine what to do next (file or dir?)
If ($DirBool) {
# Recursively traverse sub-directories
$FileTree += ,(Get-FtpDirectory "$($Directory)$($CurFile)/")
}
Else {
# Add the output to the file tree
$FileTree += ,"$($Directory)$($CurFile)"
}
}
if (!$FileTree -eq $null) {
$Verified = "Y"
Return $FileTree
}
else {
$Verified = "Failed"}
}
# Update data record field "Verified" in CSV '$data' file with status value[Y,Failed,Other]
There is where the import of the FTP login ID and Password should start
#Column Names [Connection,Name,Path,Password,Phonetic,Client Name,Client ID,Verified]
#Variables
$AllName = #()
$AllPassword = #()
$AllVerified = #()
$data = #()
$FTPLogon = #()
$FTPPass = #()
$response = #()
$GetFTP = Get-FtpDirectory
# Import CSV File "FTP_Account_List_with_Password_and_Path.csv"
$data = Import-Csv C:\Users\mgoeres\Desktop\FTP_Account_List_with_Password_and_Path.csv | where-object {$_.Connection -eq "FTP"}
# This import allows key columns to be referenced as variables.
#Import-Csv C:\Users\mgoeres\Desktop\FTP_Account_List_with_Password_and_Path.csv | ForEach-Object {
#$AllName += $_.Name
#$AllPassword += $_.Password
#$AllVerified += $_.Verified}
# Simple test to see if CSV was imported into '$data' and column headers are working as variables
$data|select-object Name,Connection,Verified
Write-Host "CSV File Opened and Loaded"
## Select (next)record from '$data' and store correlating values in '$UserName' and '$Password'
# .EXAMPLE '$data[0].Password' <= First records Password value.
# .EXAMPLE '$data[1].Name' <= Second records Name value.
for ($i=0; $i -lt $data.count; $i++){
$FTPLogon = $data[$i].Name
$FTPPass = $data[$i].Password
$Verified = $data[$i].Verified }
if ($Verifed -eq $null){ $GetFTP }
else ($i++){ $GetFTP }

PowerShell Error: Import-CSV Cannot Open File

My PowerShell code is tripping some strange error codes, which I have thoroughly searched for to no avail.
I am trying to create a script that calculates some basic statistics for a number of CSV files, which seems to work until I try to create and populate new columns.
The error code I'm getting is:
Import-Csv: Cannot open file "C:\zMFM\Microsoft.Powershell.Commands.GenericMeasureInfo".
At C:\zMFM\StatsDebug.ps1:46 Char:21
+$STATS2 = Import-CSV <<< $STATS
+CategoryInfo : OpenError (:) [Import-Csv], FileNotFoundException
+FullyQualifiedErrorId : FileOpenFailure.Microsoft.Powershell.Commands.ImportCsvCommand at C:\zMFM\statsdebug.ps1:55 char:20
That's followed by an error using a null expression, but I assume fixing this problem with Import-Csv will in turn fix that error. Here's my code, any help would be great, thanks!
$i = 1
#$colSTDDEV = New-Object System.Data.DataColumn StdDev,([double])
$colVZA = New-Object System.Data.DataColumn VZA,([double])
#$colVAZ = New-Object System.Data.DataColumn VAZ,([double])
While ($i -le 211) {
#Set the variable to the filename with the iteration number
$filename = "c:\zMFM\z550Output\20dSummer\fixed20dSum550Output$i.csv"
#Check to see if that a file with $filename exists. If not, skip to the next iteration of $i. If so, run the code to collect the
statistics for each variable and output them each to a different file
If (Test-Path $filename) {
#Calculate the Standard Deviation
#First get the average of the values in the column
$STDEVInputFile = Import-CSV $filename
#Find the average and count for column 'td'
$STDEVAVG = $STDEVInputFile | Measure-Object td -Average | Select Count, Average
$DevMath = 0
# Sum the squares of the differences between the mean and each value in the array
Foreach ($Y in $STDEVInputFile) {
$DevMath += [math]::pow(($Y.Average - $STDEVAVG.Average), 2)
#Divide by the number of samples minus one
$STDEV = [Math]::sqrt($DevMath / ($STDEVAVG.Count-1))
}
#Calculate the basic statistics for column 'td' with the MEASURE-OBJECT cmdlet
$STATS = Import-CSV $Filename |
Measure-Object td -ave -max -min
#Append the standard deviation variable to the statistics array and add the value
$STATS2 = Import-CSV $Stats
$VZA = $filename.VZA
#$VAZ = $filename.VAZ #COMMENTED FOR DEBUGING
#$STATS2.Columns.Add($colSTDDEV) #COMMENTED FOR DEBUGING
#$STATS2[0].StdDev = $STDEV #COMMENTED FOR DEBUGING
$STATS2.Columns.Add($colVZA)
$STATS2[0].VZA = $VZA
#$STATS2.Columns.Add($colVAZ) #COMMENTED FOR DEBUGING
#$STATS2[0].VZA = $VAZ #COMMENTED FOR DEBUGGING
#Export the $STATS file containing everything you need in the correct folder
Export-CSV -notype "c:\zMFM\z550Output\20dSummer\20dSum550Statistics.csv"
}
$i++
}
$STDEVInputFile = Import-CSV $filename
...
$STATS = Import-CSV $Filename |
Measure-Object td -ave -max -min
# $STATS here will be type [GenericMeasureInfo], so you cannot use this as a filename.
$STATS2 = Import-CSV $Stats
# Are you trying to import $filename a third time here? You can't use $Stats because it's a [GenericMeasureInfo] object, not a [string].
Based on your code, it looks like you're trying to import a CSV with a filename of type [Microsoft.PowerShell.Commands.GenericMeasureInfo]. Filenames are strings. Also, why are you importing $filename twice? Would recommend importing it once, then operating off of the variable you saved it to.
Would also recommend changing your while loop to 1..211 | ForEach-Object. That way you won't have to worry about whether $i is less than or equal to your static number. Not a huge deal, but would make the code a little more readable.
TL;DR
Import-CSV $Stats is the problem. $Stats is not a valid filename, so you can't open/import it.

Newly Created Column is Null.... Why?

I'm trying to do a simple task in PowerShell where some basic statistics are calculated for a number of columns in a CSV file. I'm nearly done, but I keep getting an error where new columns I create are coming up as being Null. I cannot see where I am going wrong, here.
Specifically, the line of code causing the error is
$STATS2.Columns.Add($colVZA) |
The tables created when importing $filename do have columns named VZA, VAZ, etc., so that's not the problem.
It seems like adding and populating columns should be a simple task, so I'm sure I'm missing something simple here. Here's my code:
#######################
function Get-Type
{
param($type)
$types = #(
'System.Boolean',
'System.Byte[]',
'System.Byte',
'System.Char',
'System.Datetime',
'System.Decimal',
'System.Double',
'System.Guid',
'System.Int16',
'System.Int32',
'System.Int64',
'System.Single',
'System.UInt16',
'System.UInt32',
'System.UInt64')
if ( $types -contains $type ) {
Write-Output "$type"
}
else {
Write-Output 'System.String'
}
} #Get-Type
#######################
<#
.SYNOPSIS
Creates a DataTable for an object
.DESCRIPTION
Creates a DataTable based on an objects properties.
.INPUTS
Object
Any object can be piped to Out-DataTable
.OUTPUTS
System.Data.DataTable
.EXAMPLE
$dt = Get-psdrive| Out-DataTable
This example creates a DataTable from the properties of Get-psdrive and assigns output to $dt variable
.NOTES
Adapted from script by Marc van Orsouw see link
Version History
v1.0 - Chad Miller - Initial Release
v1.1 - Chad Miller - Fixed Issue with Properties
v1.2 - Chad Miller - Added setting column datatype by property as suggested by emp0
v1.3 - Chad Miller - Corrected issue with setting datatype on empty properties
v1.4 - Chad Miller - Corrected issue with DBNull
v1.5 - Chad Miller - Updated example
v1.6 - Chad Miller - Added column datatype logic with default to string
v1.7 - Chad Miller - Fixed issue with IsArray
.LINK
http://thepowershellguy.com/blogs/posh/archive/2007/01/21/powershell-gui-scripblock-monitor-script.aspx
#>
function Out-DataTable
{
[CmdletBinding()]
param([Parameter(Position=0, Mandatory=$true, ValueFromPipeline = $true)] [PSObject[]]$InputObject)
Begin
{
$dt = new-object Data.datatable
$First = $true
}
Process
{
foreach ($object in $InputObject)
{
$DR = $DT.NewRow()
foreach($property in $object.PsObject.get_properties())
{
if ($first)
{
$Col = new-object Data.DataColumn
$Col.ColumnName = $property.Name.ToString()
if ($property.value)
{
if ($property.value -isnot [System.DBNull]) {
$Col.DataType = [System.Type]::GetType("$(Get-Type $property.TypeNameOfValue)")
}
}
$DT.Columns.Add($Col)
}
if ($property.Gettype().IsArray) {
$DR.Item($property.Name) =$property.value | ConvertTo-XML -AS String -NoTypeInformation -Depth 1
}
else {
$DR.Item($property.Name) = $property.value
}
}
$DT.Rows.Add($DR)
$First = $false
}
}
End
{
Write-Output #(,($dt))
}
$i = 1
While ($i -le 211) {
#Set the variable to the filename with the iteration number
$filename = "c:\zMFM\z550Output\20dSummer\fixed20dSum550Output$i.csv"
#Check to see if that a file with $filename exists. If not, skip to the next iteration of $i. If so, run the code to collect the statistics for each variable and output them each to a different file
If (Test-Path $filename) {
#Calculate the Standard Deviation
#First get the average of the values in the column
$STDEVInputFile = Import-CSV $filename
#Find the average and count for column 'td'
$STDEVAVG = $STDEVInputFile | Measure-Object td -Average | Select Count, Average
$DevMath = 0
# Sum the squares of the differences between the mean and each value in the array
Foreach ($Y in $STDEVInputFile) {
$DevMath += [math]::pow(($Y.Average - $STDEVAVG.Average), 2)
#Divide by the number of samples minus one
$STDEV = [Math]::sqrt($DevMath / ($STDEVAVG.Count-1))
}
#Calculate the basic statistics for column 'td' with the MEASURE-OBJECT cmdlet
$STATS = Import-CSV $Filename |
Measure-Object td -ave -max -min |
#Export the statistics as a CSV
Export-CSV -notype "c:\zMFM\z550Output\20dSummer\tempstats$i.csv"
$GetColumns = Import-CSV $filename
#Append the standard deviation variable to the statistics table and add the value
$STATS2 = Import-CSV "c:\zMFM\z550Output\20dSummer\tempstats$i.csv"
$StatsTable = Get-PSDrive | Out-DataTable
#$colSTDDEV = New-Object System.Data.DataColumn StdDev,([double])
$colVZA = New-Object System.Data.DataColumn VZA,([double])
#$colVAZ = New-Object System.Data.DataColumn VAZ,([double])
$colVZA = $GetColumns[0].VZA
#$colVAZ = $GetColumns[0].VAZ #COMMENTED FOR DEBUGGING
#$colSTDDEV = $STDEV
#$StatsTable.Columns.Add($colSTDDEV) #COMMENTED FOR DEBUGGING
#$StatsTable[0].StdDev = $STDEV #COMMENTED FOR DEBUGGING
$StatsTable.Columns.Add($colVZA) |
#$StatsTable[0].VZA = $VZA
#$StatsTable.Columns.Add($colVAZ) #COMMENTED FOR DEBUGGING
#$StatsTable[0].VZA = $VAZ #COMMENTED FOR DEBUGGING
#Export the $STATS file containing everything you need in the correct folder
Export-CSV -notype "c:\zMFM\z550Output\20dSummer\20dSum550Statistics.csv"
}
$i++
}
Even though each object in $STATS2 have the same properties, the $STATS2 object itself is just a simple array, an unstructured list of objects - it doesn't have a Columns property with an Add() method:
$STATS2.Colums.Add($colVZA)
^ ^ ^
[array] | |
$null |
this fails
You can convert the array you get from Import-Csv from an array to a DataTable object (which has Columns) by inspecting each property in the first object in the array, like in the Out-DataTable sample on the technet script gallery