Somewhat confused as what I need to use to acheive my goal. Here is what I need to do:
Recursively, get a list of files
Get the filename, lastwritedate, owner and append to an array
When the array reaches 500, output it to a csv filename.csv. Have a file for each 500 line output
Continue on, outputting files with details (500 lines) until end
I'm not conceptually grasping if I need a recursive function or can do some tricky while/do/for loop work.
Here is what I have so far:
$array = #()
$files = ls c:\users –recurse
$counter = 0
$filename = 0
foreach ($file in $files){
if($counter -ne 500){
write-host "Doing loop, counter is " $counter
$owner = Get-Acl $file.FullName
$data = new-object -type PSObject
$data | add-member -membertype noteproperty -name "File" -value $file.FullName
$data | add-member -membertype noteproperty -name "LastWriteTime" -value $file.LastWriteTime
$data | add-member -membertype noteproperty -name "Created" -value $file.CreationTime
$data | add-member -membertype noteproperty -name "Owner" -value $file.Owner
$array += $data
$counter ++
else($counter -eq 500){
$counter = 0
$array = #()
$array | export-csv "c:\temp\filelist\file-$filename.csv"
$filename ++
}
}
}
I've actually tried a bunch of different things, loop types etc. I just cant quiet figure out how to get the script to "goto start of for loop" as my understanding isn't there yet.
The reason I'm limiting to 500 lines is because this will run on a fileserver with millions of files, and a simple get-childitem wasn't quiet handling it gracefully.
I'm happy to not be given the direct answer, but given some hints as to what I need to do would be great (this isn't homework, I like learning powershell ha :))
I think I would nest a For loop within another For loop do to what you want. Something like:
For($x=0;$x -lt $Files.count;$x=$x+500){
$Array=#()
For($y=0;$y -lt 500; $y++){
$Array += $Files[($x+$y)] | Select FullName,LastWriteTime,CreationTime,#{l='Owner';e={($_|Get-ACL).Owner}}
}
$Array|Export-CSV C:\Path\To\Export-$x.csv -NoType
}
Related
I have a piece of code that works but I want to know if there is a better way to do it. I could not find anything related so far. Here are the facts:
I have an object with n properties.
I want to convert this object to JSON using (ConvertTo-Json).
I don't want to include in the JSON those object properties that are not valued.
Building the object (not really important):
$object = New-Object PSObject
Add-Member -InputObject $object -MemberType NoteProperty -Name TableName -Value "MyTable"
Add-Member -InputObject $object -MemberType NoteProperty -Name Description -Value "Lorem ipsum dolor.."
Add-Member -InputObject $object -MemberType NoteProperty -Name AppArea -Value "UserMgmt"
Add-Member -InputObject $object -MemberType NoteProperty -Name InitialVersionCode -Value ""
The line that I need improvements (to filter out the non-valued properties and not include them in the JSON)
# So I want to 'keep' and deliver to the JSON only the properties that are valued (first 3).
$object | select -Property TableName, Description, AppArea, InitialVersion | ConvertTo-Json
What this line delivers:
Results:
{
"TableName": "MyTable",
"Description": "Lorem ipsum dolor..",
"AppArea": "UserMgmt",
"InitialVersion": null
}
What I want to obtain:
{
"TableName": "MyTable",
"Description": "Lorem ipsum dolor..",
"AppArea": "UserMgmt"
}
What I've tried and works, but I don't like it since I have much more properties to handle:
$JSON = New-Object PSObject
if ($object.TableName){
Add-Member -InputObject $JSON -MemberType NoteProperty -Name TableName -Value $object.TableName
}
if ($object.Description){
Add-Member -InputObject $JSON -MemberType NoteProperty -Name Description -Value $object.Description
}
if ($object.AppArea){
Add-Member -InputObject $JSON -MemberType NoteProperty -Name AppArea -Value $object.AppArea
}
if ($object.InitialVersionCode){
Add-Member -InputObject $JSON -MemberType NoteProperty -Name InitialVersionCode -Value $object.InitialVersionCode
}
$JSON | ConvertTo-Json
Something like this?
$object = New-Object PSObject
Add-Member -InputObject $object -MemberType NoteProperty -Name TableName -Value "MyTable"
Add-Member -InputObject $object -MemberType NoteProperty -Name Description -Value "Lorem ipsum dolor.."
Add-Member -InputObject $object -MemberType NoteProperty -Name AppArea -Value "UserMgmt"
Add-Member -InputObject $object -MemberType NoteProperty -Name InitialVersionCode -Value ""
# Iterate over objects
$object | ForEach-Object {
# Get array of names of object properties that can be cast to boolean TRUE
# PSObject.Properties - https://msdn.microsoft.com/en-us/library/system.management.automation.psobject.properties.aspx
$NonEmptyProperties = $_.psobject.Properties | Where-Object {$_.Value} | Select-Object -ExpandProperty Name
# Convert object to JSON with only non-empty properties
$_ | Select-Object -Property $NonEmptyProperties | ConvertTo-Json
}
Result:
{
"TableName": "MyTable",
"Description": "Lorem ipsum dolor..",
"AppArea": "UserMgmt"
}
I have the following function in my profile for this purpose. Advantage: I can pipe a collection of objects to it and remove nulls from all the objects on the pipeline.
Function Remove-Null {
[cmdletbinding()]
param(
# Object to remove null values from
[parameter(ValueFromPipeline,Mandatory)]
[object[]]$InputObject,
#By default, remove empty strings (""), specify -LeaveEmptyStrings to leave them.
[switch]$LeaveEmptyStrings
)
process {
foreach ($obj in $InputObject) {
$AllProperties = $obj.psobject.properties.Name
$NonNulls = $AllProperties |
where-object {$null -ne $obj.$PSItem} |
where-object {$LeaveEmptyStrings.IsPresent -or -not [string]::IsNullOrEmpty($obj.$PSItem)}
$obj | Select-Object -Property $NonNulls
}
}
}
Some examples of usage:
$AnObject = [pscustomobject]#{
prop1="data"
prop2="moredata"
prop5=3
propblnk=""
propnll=$null
}
$AnObject | Remove-Null
prop1 prop2 prop5
----- ----- -----
data moredata 3
$ObjList =#(
[PSCustomObject]#{
notnull = "data"
more = "sure!"
done = $null
another = ""
},
[PSCustomObject]#{
notnull = "data"
more = $null
done = $false
another = $true
}
)
$objList | Remove-Null | fl #format-list because the default table is misleading
notnull : data
more : sure!
notnull : data
done : False
another : True
beatcracker's helpful answer offers an effective solution; let me complement it with a streamlined version that takes advantage of PSv4+ features:
# Sample input object
$object = [pscustomobject] #{
TableName = 'MyTable'
Description = 'Lorem ipsum dolor...'
AppArea = 'UserMgmt'
InitialVersionCode = $null
}
# Start with the list of candidate properties.
# For simplicity we target *all* properties of input object $obj
# but you could start with an explicit list as wellL
# $candidateProps = 'TableName', 'Description', 'AppArea', 'InitialVersionCode'
$candidateProps = $object.psobject.properties.Name
# Create the filtered list of those properties whose value is non-$null
# The .Where() method is a PSv4+ feature.
$nonNullProps = $candidateProps.Where({ $null -ne $object.$_ })
# Extract the list of non-null properties directly from the input object
# and convert to JSON.
$object | Select-Object $nonNullProps | ConvertTo-Json
I made my own modified version of batmanama's answer that accepts an additional parameter, letting you remove elements that are also present in the list present in that parameter.
For example:
Get-CimInstance -ClassName Win32_UserProfile |
Remove-Null -AlsoRemove 'Win32_FolderRedirectionHealth' | Format-Table
I've posted a gist version including PowerShell documentation as well.
Function Remove-Null {
[CmdletBinding()]
Param(
# Object from which to remove the null values.
[Parameter(ValueFromPipeline,Mandatory)]
$InputObject,
# Instead of also removing values that are empty strings, include them
# in the output.
[Switch]$LeaveEmptyStrings,
# Additional entries to remove, which are either present in the
# properties list as an object or as a string representation of the
# object.
# I.e. $item.ToString().
[Object[]]$AlsoRemove = #()
)
Process {
# Iterate InputObject in case input was passed as an array
ForEach ($obj in $InputObject) {
$obj | Select-Object -Property (
$obj.PSObject.Properties.Name | Where-Object {
-not (
# If prop is null, remove it
$null -eq $obj.$_ -or
# If -LeaveEmptyStrings is not specified and the property
# is an empty string, remove it
(-not $LeaveEmptyStrings.IsPresent -and
[string]::IsNullOrEmpty($obj.$_)) -or
# If AlsoRemove contains the property, remove it
$AlsoRemove.Contains($obj.$_) -or
# If AlsoRemove contains the string representation of
# the property, remove it
$AlsoRemove.Contains($obj.$_.ToString())
)
}
)
}
}
}
Note that the process block here automatically iterates a pipeline object, so the ForEach will only iterate more than once when an item is either explicitly passed in an array—such as by wrapping it in a single element array ,$array—or when provided as a direct argument, such as Remove-Null -InputObject $(Get-ChildItem).
It's also worth mentioning that both mine and batmanama's functions will remove these properties from each individual object. That is how it can properly utilize the PowerShell pipeline. Furthermore, that means that if any of the objects in the InputObject have a property that does not match (e.g. they are not null), an output table will still show that property, even though it has removed those properties from other items that did match.
Here's a simple example showing that behavior:
#([pscustomobject]#{Number=1;Bool=$true};
[pscustomobject]#{Number=2;Bool=$false},
[pscustomobject]#{Number=3;Bool=$true},
[pscustomobject]#{Number=4;Bool=$false}) | Remove-Null -AlsoRemove $false
Number Bool
------ ----
1 True
2
3 True
4
I am trying to get the list of machines which are in particular state (Saved, Running, Stopped). I am passing the state of the machine as an argument in a function.
Function global:Resource-Summary
{
Param(
[parameter(mandatory=$true)] $ProgramName,
[parameter(mandatory=$true)] $ServerName
)
PROCESS
{
Foreach ($Server in $ServerName)
{
Invoke-Command -ComputerName $Server -ScriptBlock {
$VMs = Get-VM
$colVMs = #()
foreach ($VM in $VMs)
{
$objVM = New-Object System.Object
$objVM | Add-Member -MemberType NoteProperty -Name VMName -Value $VM.VMName
$objVM | Add-Member -MemberType NoteProperty -Name VMNotes -Value $VM.Notes
$objVM | Add-Member -MemberType NoteProperty -Name VMState -Value $VM.State
$colVMs += $objVM
}
$a = #{Expression={$_.VMName};Label='VM Name'}, `
#{Expression={$_.VMNotes};Label='VM Description'}, `
#{Expression={$_.VMState};Label='State'}
"Program Name : $ProgramName"
$colVMs |Where-Object {($_.VMState -eq '$ProgramName')} | Format-Table $a -AutoSize
} -ArgumentList $ProgramName
}
}
}
When I run Resource-Summary -ProgramName Running -ServerName Demo
I do not get any value.
When I replace $ProgramName with RUNNING I get the expected output.
For reference, see e.g. this post on how Pass arguments to a scriptblock in powershell
.
The problem in your script is how you call the script block, this is explained in more detail in the link above, but you need to pass any "external" input to it the same way as if you'd call it like a function.
You are doing this partially correctly, you are using the -ArgumentList parameter to send $ProgramName to the scriptslock but you haven't specified in the scriptblock how to access it.
For example, check out
Invoke-Command -ArgumentList "Application" -ScriptBlock {
param($log)
Get-EventLog $log
}
Here -ArgumentList contains the input, and inside the scriptblock, $log is assigned its value.
Updating your script to take that into account:
Function global:Resource-Summary
{
Param(
[parameter(mandatory=$true)] $ProgramName,
[parameter(mandatory=$true)] $ServerName
)
PROCESS
{
Foreach ($Server in $ServerName)
{
Invoke-Command -ComputerName $Server -ScriptBlock {
param($name)
$VMs = Get-VM
$colVMs = #()
foreach ($VM in $VMs)
{
$objVM = New-Object System.Object
$objVM | Add-Member -MemberType NoteProperty -Name VMName -Value $VM.VMName
$objVM | Add-Member -MemberType NoteProperty -Name VMNotes -Value $VM.Notes
$objVM | Add-Member -MemberType NoteProperty -Name VMState -Value $VM.State
$colVMs += $objVM
}
$a = #{Expression={$_.VMName};Label='VM Name'}, `
#{Expression={$_.VMNotes};Label='VM Description'}, `
#{Expression={$_.VMState};Label='State'}
"Program Name : $ProgramName"
$colVMs |Where-Object {($_.VMState -eq "$name") | Format-Table $a -AutoSize
}
} -ArgumentList $ProgramName
}
}
}
Also, in my opinion, the -ArgumentList is on the wrong line, it needs to be after the following closing bracket, one line done. This way it's on the same cmdlet as Invoke-Command and the scriptblock.
A couple of things jump out:
If you have a variable inside single quotes, it won't be automatically resolved by PowerShell, so in this section, you are actually comparing the state of the VM to the string $ProgramName and not it's value:
$_.VMState -eq '$ProgramName'
Try changing to double quotes, or none at all.
Also, check the bracketing - you're missing the closing one for Where-Object.
This question already has answers here:
Can Powershell Run Commands in Parallel?
(10 answers)
Closed 6 years ago.
I have 100,000 list of servers from the text file (serverlist.txt)
When I run in one shot it will burst my memory and cpu and the time took longer (about 3 days)to complete the scanning for DNSlookup.
I tried to split the file that contain 20k list of servers below and can be completed to scan up to 10mins for each file.
serverlist1.txt
serverlist2.txt
serverlist3.txt
serverlist4.txt
serverlist5.txt
$objContainer = #()
$values = #()
$domains = Get-Content -path "serverlist1.txt"
$named = 0
$timestamp= get-date
$domains | ForEach-Object {
$domain = $_
nslookup $domain 2>&1 | ForEach-Object {
if ($_ -match '^Name:\s*(.*)$') {
$values += $matches[1]
$named = 1;
} elseif (($_ -match '^.*?(\d*\.\d*\.\d*\.\d*)$') -and ($named -eq 1)) {
$values += $matches[1]
} elseif ($_ -match '^Aliases:\s*(.*)$') {
$values += $matches[1]
}
}
$obj = New-Object -TypeName PSObject
#$obj | Add-Member -MemberType NoteProperty -name 'Domain' -value $domain
$obj | Add-Member -MemberType NoteProperty -name 'Name' -value $values[0]
$obj | Add-Member -MemberType NoteProperty -name 'IP Address' -value $values[1]
$obj | Add-Member -MemberType NoteProperty -name 'Alias' -value $values[2]
$obj | Add-Member -MemberType NoteProperty -name 'Timestamp' -value $timestamp
$objContainer += $obj
$values = #()
$named = 0
}
Write-Output $objContainer
$objContainer | Export-csv "dnslog_$((Get-Date).ToString('MM-dd-yyyy_hh-mm-ss')).csv" -NoTypeInformation
My question is, how to execute at once and looping the input from the text file after generate the dnslog(datetime).csv
e.g:
run the powershell script .\filename.ps1
input from serverlist1.txt
output dnslog(datetime).csv
input from serverlist2.txt
output dnslog(datetime).csv
input from serverlist3.txt
output dnslog(datetime).csv
input from serverlist4.txt
output dnslog(datetime).csv
input from serverlist5.txt
output dnslog(datetime).csv
Finish!
If i have more then 5 list of text file, it will continue to loop from the input file until completed.
Adding to Chris's answer I would also add a ReadCount flag to the Get-Content like so:
Get-Content -path "serverlist1.txt" -ReadCount 1 | % {
This will save having to read the entire file into memory.
You should consider running this a parallel batching job. Have you already tried doing so?
You can deal with the RAM busting problem by removing all those commits to memory (variable assignments and array rewriting with +=).
$timestamp = get-date
Get-Content -path "serverlist1.txt" | ForEach-Object {
$domain = $_
# You can clear this here.
$values = #()
$named = 0
# There are potentially better options than nslookup.
# Needs a bit of care to understand what's an alias here though.
# [System.Net.Dns]::GetHostEntry($domain)
# And if you don't like that, quite a few of us have written equivalent tools in PowerShell.
nslookup $domain 2>&1 | ForEach-Object {
if ($_ -match '^Name:\s*(.*)$') {
$values += $matches[1]
$named = 1;
} elseif (($_ -match '^.*?(\d*\.\d*\.\d*\.\d*)$') -and ($named -eq 1)) {
$values += $matches[1]
} elseif ($_ -match '^Aliases:\s*(.*)$') {
$values += $matches[1]
}
}
# Leave the output object in the output pipeline
# If you're running PowerShell 3 or better:
[PSCustomObject]#{
Domain = $domain
Name = $values[0]
'IP Address' = $values[1]
Alias = $values[2]
TimeStamp = $timestamp
}
# PowerShell 2 is less flexible. This or Select-Object.
#$obj = New-Object -TypeName PSObject
##$obj | Add-Member -MemberType NoteProperty -name 'Domain' -value $domain
#$obj | Add-Member -MemberType NoteProperty -name 'Name' -value $values[0]
#$obj | Add-Member -MemberType NoteProperty -name 'IP Address' -value $values[1]
#$obj | Add-Member -MemberType NoteProperty -name 'Alias' -value $values[2]
#$obj | Add-Member -MemberType NoteProperty -name 'Timestamp' -value $timestamp
# To leave this in the output pipeline, uncomment this
# $obj
# No version of PowerShell needs you to do this. It's a good way to ramp up memory usage
# for large data sets.
# $objContainer += $obj
} | Export-Csv "dnslog_$(Get-Date -Format 'MM-dd-yyyy_hh-mm-ss').csv" -NoTypeInformation
I have a simple object with 1 parameter being an ArrayList of objects. I am using ConvertTo-Json to output this to Json. However even if I set -Depth 1000 I still see truncation of data.
Structure is:
Object
Property
Property - ArrayList of Object2.
Object 2 is a simple collection of properties.
The output I see is:
{
"CheckDate": "03 February 2016 10:12:30",
"Versions": [
{
},
{
}
]
}
Calling convert on the ArrayList directly all the data is shown. It would appear as if the -Depth argument is not being honored and is stuck at 2.
edit: Code to create object
$returnValue = New-Object System.Object
$returnValue | Add-Member -type NoteProperty -name CheckDate -value (Get-Date).DateTime
$versions = New-Object System.Collections.ArrayList
# This bit is in a loop.
$app = New-Object System.Object
$app | Add-Member -type NoteProperty -Name Name -Value $name
$app | Add-Member -type NoteProperty -Name Version -Value $version
$versions.Add($app)
# Back out of the loop.
$returnValue | Add-Member -type NoteProperty -name Versions -value $versions
Use PSObject instead of System.Object. Unfortunately, I cannot provide any details, it is some internal "magic" of ConvertTo-Json. Interestingly, it is enough to use PSObject instead of the second System.Object.
I'm running into this over and over in my script. I have this line of code:
$Errors = Get-DeploymentErrors $uniqueID
When this runs, $Errors is assigned the result of Get-DeploymentErrors and the value of $uniqueID. I just want $Errors to be assigned the result of Get-DeploymentErrors.
Here is the Get-DeploymentErrors function:
Function Get-DeploymentErrors($uniqueID)
{
$Errors = #()
$conn = New-Object -TypeName System.Data.SqlClient.SqlConnection
$conn.ConnectionString = 'removed connection string'
$cmd = New-Object -TypeName System.Data.SqlClient.SqlCommand
$cmd.Connection = $conn
$cmd.CommandText = "removed sql statement"
$cmd.Parameters.AddWithValue("#uniqueID", $uniqueID)
$conn.Open()
$reader = $cmd.ExecuteReader()
if($reader.HasRows)
{
While ($reader.Read())
{
$error = New-Object -TypeName PSObject
$error | Add-Member -MemberType NoteProperty -Name StepID -Value $reader["StepID"]
$error | Add-Member -MemberType NoteProperty -Name DeploymentID -Value $reader["DeploymentID"]
$error | Add-Member -MemberType NoteProperty -Name MessageID -Value $reader["MessageID"]
$error | Add-Member -MemberType NoteProperty -Name Severity -Value $reader["Severity"]
$error | Add-Member -MemberType NoteProperty -Name Message -Value $reader["Message"]
$error | Add-Member -MemberType NoteProperty -Name StepName -Value $reader["StepName"]
$error | Add-Member -MemberType NoteProperty -Name CurrentStep -Value $reader["CurrentStep"]
$error | Add-Member -MemberType NoteProperty -Name TotalSteps -Value $reader["TotalSteps"]
$error | Add-Member -MemberType NoteProperty -Name CurrentTime -Value $reader["CurrentTime"]
$Errors += $error
}
}
return $Errors
}
$cmd.Parameters.AddWithValue() echoes the added parameter, and PowerShell functions return the entire non-captured output on the success output stream, not just the argument of the return keyword.
Quoting from about_Return (emphasis mine):
SHORT DESCRIPTION
Exits the current scope, which can be a function, script, or script block.
LONG DESCRIPTION
The Return keyword exits a function, script, or script block. It can be used to exit a scope at a specific point, to return a value, or to indicate that the end of the scope has been reached.
Users who are familiar with languages like C or C# might want to use the Return keyword to make the logic of leaving a scope explicit.
In Windows PowerShell, the results of each statement are returned as output, even without a statement that contains the Return keyword. Languages like C or C# return only the value or values that are specified by the Return keyword.
Use any of the following methods to suppress the undesired output:
[void]$cmd.Parameters.AddWithValue("#uniqueID", $uniqueID)
$cmd.Parameters.AddWithValue("#uniqueID", $uniqueID) | Out-Null
$cmd.Parameters.AddWithValue("#uniqueID", $uniqueID) > $null
$param = $cmd.Parameters.AddWithValue("#uniqueID", $uniqueID)