Kind of have a strange problem. I have a large JSON file that needs to be processed. Based on another question I need to stream the file since it will otherwise gets me problems because of memory: JSON Powershell memory issue
What I have is this:
get-content -Path largefile.json | ForEach-Object {
$row = $_ = $_.TrimStart('[').TrimEnd(']')
if ($_) { $_ | Out-String | ConvertFrom-Json }
New-Item -Path $($Row.Id).txt
Set-Content -Path $($Row.Id).txt -Value ($row.Body)
}
I can easily do $row to publish the last processed row in the Largefile.json. I want to create a file with the name of the Id in the row that is currently processed and add the body column to the file. But when I want to show a specific column using $row.Id, unfortunately this shows up empty.
The structure of the Largefile.json is as followed:
[{"Id":"1","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data1"}
{"Id":"2","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data2"}
{"Id":"3","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data3"}
{"Id":"4","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data4"}
{"Id":"5","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"data5"}
]
The end result should be that I have 5 files:
1.txt - Value inside the file should be: data1
2.txt - Value inside the file should be: data2
3.txt - Value inside the file should be: data3
4.txt - Value inside the file should be: data4
5.txt - Value inside the file should be: data5
I use Powershell 7.1.3
Is there any way that I can use $row.Id and $row.ParentId just like a regular ForEach would do?
thanks for your help.
It seems to me that this is what you're looking for:
Get-Content largefile.json | ForEach-Object {
$row = $_.TrimStart('[').TrimEnd(']') | ConvertFrom-Json
if ($null -ne $row) {
Set-Content -Path ($row.Id) -Value ($row.Body)
}
}
I am still not sure what you expect as an outcome.
But I think you want to do this:
#'
[{"Id":"1","ParentId":"parent1","Name":"1.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data1"}
{"Id":"2","ParentId":"parent2","Name":"2.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data2"}
{"Id":"3","ParentId":"parent3","Name":"3.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data3"}
{"Id":"4","ParentId":"parent4","Name":"4.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data4"}
{"Id":"5","ParentId":"parent5","Name":"5.txt","OwnerId":"owner","CreatedDate":"date","Body":"Data5"}
]
'# | Set-Content .\largefile.json
Get-Content .\largefile.json | ForEach-Object {
$_ = $_.TrimStart('[').TrimEnd(']')
If ($_) {
$Row = ConvertFrom-Json $_
Set-Content -Path ".\$($Row.Name)" -Value $Row.Body
}
}
The question has many errors. Assuming the json has the missing commas in, I would do it this way, if I understand the question. This should work with the new updates to the question. I also have a more unusual solution involving streaming json with jq here: Iterate though huge JSON in powershell Json streaming support may be added later: ConvertFrom-JSON high memory consumption #7698
[{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"},
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"},
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"},
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"},
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
]
get-content -Path largefile.json | ForEach-Object {
$_ = $_.TrimStart('[').TrimEnd(']').TrimEnd(',')
if ($_) {
$row = $_ | ConvertFrom-Json
Set-Content -Path ($Row.Id + '.txt') -Value $row.Body
}
}
get-content ID.txt
*******
As others already explained, your json example is invalid.
However, since this is a huge file to process, you can use switch for this.
switch -Regex -File D:\Test\largefile.json {
'"Id":"(\d+)".*"Body":"(\w+)"' {
Set-Content -Path ('D:\Test\{0}.txt' -f $matches[1]) -Value $matches[2]
}
}
Results using your example would be 5 files called 1.txt .. 5.txt, each having a single line data1 .. data5
This is my PowerShell which is calling an API and returning a JSON response.
$output = Get-SurveyParticipents `
-url "https://orxsurveys.limequery.com/admin/remotecontrol" `
-session $sessionKey `
-id "519965" `
-start "0" `
-limit "2" `
-unused $False `
-attributes ["completed", "usesleft"]
($output | Export-Csv -NoTypeInformation ./testtt.csv)
Write-host $output produces:
#{tid=6; token=35ddmyQTlNpzLat; participant_info=} #{tid=7; token=nQ_S838LjYT4mR6; participant_info=}
Export-CSV produces:
This is what I need to produce from export-csv:
Can anybody please point me in the right direction to transforming the 'participant_info' into valid json for the CSV export? - As you can tell I have little expierence with PowerShell other then using it for SharePoint. Thank you!
Your goal is to output an object with a custom set of properties (because it differs from the original object). This can be done with Select-Object and calculated properties.
$output |
Select-Object tid,token,
#{n='Firstname';e={$_.participant_info.Firstname}},
#{n='Lastname';e={$_.participant_info.Lastname}},
#{n='Email';e={$_.participant_info.Email}} |
Export-CSV testtt.csv -NoType
It looks like the way I am expecting this to work doesn't. I want multiple objects returned, but it seems to be returning just one. It is beyond me how I do it.
A very simple JSON file:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"storageAccountName": {
"value": "sa01"
},
"virtualNetworkName": {
"value": "nvn01"
}
}
}
I want to dynamically add the parameters and their values into a nice pscustomobject (that would look like the following with the above data):
ParameterName | Value
===========================
storageAccountName | sa01
virtualNetworkName | nvn01
What I don't understand is why the following returns one object:
$TemplateParametersFile = "C:\Temp\deploy-Project-Platform.parameters.json"
$content = Get-Content $TemplateParametersFile -Raw
$JsonParameters = ConvertFrom-Json -InputObject $content
$JsonParameters.parameters | Measure-Object
Whilst writing this, I eventually found a solution that get's what I want, which I'll post in the answer section. Feel free to school me and improve...
I would do things a little differently, skipping the hashtable, and using the hidden PSObject property. So, picking up after you have the JSON data stored in $content, I would do something like this:
#Convert JSON file to an object
$JsonParameters = ConvertFrom-Json -InputObject $content
#Create new PSObject with no properties
$oData = New-Object PSObject
#Loop through properties of the $JsonParameters.parameters object, and add them to the new blank object
$JsonParameters.parameters.psobject.Properties.Name |
ForEach{
Add-Member -InputObject $oData -NotePropertyName $_ -NotePropertyValue $JsonParameters.parameters.$_.Value
}
$oData
By the way, it had issues converting the JSON you posted, I had to add quotes around the two values, such as "value": "sa01".
Using the same JSON file as shown above:
<#
# Read in JSON from file on disk
$TemplateParametersFile = "C:\Temp\deploy-Project-Platform.parameters.json"
$content = Get-Content $TemplateParametersFile -Raw
#>
#Retrieve JSON file from Azure storage account.
$TemplateParametersFile = "https://{storageAccountName}.blob.core.windows.net/{SomeContainer}/deploy-Project-Platform.parameters.json"
$oWc = New-Object System.Net.WebClient
$webpage = $oWc.DownloadData($TemplateParametersFile)
$content = [System.Text.Encoding]::ASCII.GetString($webpage)
#Convert JSON file to an object (IMHO- Sort of!)
$JsonParameters = ConvertFrom-Json -InputObject $content
#Build hashtable - easier to add new items - the whole purpose of this script
$oDataHash = #{}
$JsonParameters.parameters | Get-Member -MemberType NoteProperty | ForEach-Object{
$oDataHash += #{
$_.name = $JsonParameters.parameters."$($_.name)" | Select -ExpandProperty Value
}
}
#Example: adding a single item to the hashtable
$oDataHash.Add("VirtualMachineName","aDemoAdd")
#Convert hashtable to pscustomobject
$oData = New-Object -TypeName PSCustomObject
$oData | Add-Member -MemberType ScriptMethod -Name AddNote -Value {
Add-Member -InputObject $this -MemberType NoteProperty -Name $args[0] -Value $args[1]
}
$oDataHash.Keys | Sort-Object | ForEach-Object{
$oData.AddNote($_,$oDataHash.$_)
}
$oData
And the result:
storageAccountName VirtualMachineName virtualNetworkName
------------------ ------------------ ------------------
sa01 aDemoAdd nvn01
Agreed, the question asked for a Parameter / Value pair, and this results in the parameter's name being assigned as the noteproperty, but I think it will be easier to use it this way. Of course, $oDataHash returns it as a Key/value pair.
This script also pulls the JSON file directly from an Azure storage account. No need to save to disk. If you want to save to disk, change $oWc.DownloadData() to $oWc.DownloadFile() . The commented bit at the top, reads from disk.
I am sure there are much more succinct ways to achieve the same result, and I'd love to here them. For me, at the moment this works.
I'm trying to import users into an active directory using a CSV file and a powershell script. I create a CSV with the headers normally associated with an AD object:
mail,name,givenName,middleName,surname,company,department,title,plaintextPassword,path,description,userPrincipalName
...and filled it up.
Now I want to use Powershell's new-aduser cmmdlet to generate users for each item in this sheet - the problem I'm having is that new-aduser requires a SecureString, not just a normal string for an account's password. Skipping this conversion results in my users being created correctly, but with no passwords and their account disabled.
The command I'm using is as follows:
import-csv .\users.csv | add-member -passthru -memberType NoteProperty -value {$_ | select plaintextPassword | ConvertTo-SecureString -fromplaintext -force}
The result is user records like the following:
mail : tom.fubar#contoso.com
name : tom.fubar
givenName : Tom
middleName :
surname : Fubar
company : Contoso
department : IT
title : Technician
accountPassword : LongPasswordThatFitsADComplexityRequirements123!
path : OU=UserAccounts,OU=IT,OU=employees,DC=contoso,DC=com
description :
userPrincipalName : tom.fubar#contoso.com
encodedPassword : {$_ | select accountPassword | ConvertTo-SecureString -asplaintext -force}
The bit of code that should be evaluated for converting the plaintext password to a SecureString is being passed verbatim, rather than executed inline.
What is the proper way to force the code block to be evaluated, and use its result as the argument to New-Member -value?
Tried:
Enclosing the script block in $(...) - Results in a null NoteProperty added to the object
Replacing the {...} with $(...) - Results in a null NoteProperty added to the object
(as shown by piping the whole command to Get-Member)
Eris has posted a perfectly valid workaround, but to answer why it won't work for you, it's because $_ doesn't apply to any old script block. It's "special" and used only in certain contexts.
Additionally, -NotePropertyValue expects a static value.
Instead, you could add a ScriptProperty like so:
import-csv .\users.csv |
add-member -passthru -memberType ScriptProperty -value {$this.plaintextPassword | ConvertTo-SecureString -fromplaintext -force}
In this context for example, $_ is not even used; you have to use $this to refer to the parent object.
This does result in the script being processed every time the property is accessed though. If you don't want that, and want to do a static value assignment that's calculated per object, then you must enumerate yourself:
import-csv .\users.csv | ForEach-Object {
$val = $_ | select plaintextPassword | ConvertTo-SecureString -fromplaintext -force
$_ | add-member -passthru -memberType NoteProperty -value $val -Force
}
One solution I've found is to not bother with Add-Member, instead use a calculated property like so:
import-csv .\users.csv |
select -Property *, #{
n="encodedPassword";
e={$_.plaintestPassword | ConvertTo-SecureString -fromplaintext -force}}
(Removed broken add-member after comment from #PetSerAl)
I have a PowerShell scripts which replaces
"version" : "xxx"
with
"version" : "myBuildNumber"
Now I encountered that I have multiple of these in my file.
I only want to replace the first occurrence.
I tried already Powershell - Replace first occurrences of String but it does not work with my regex.
Here's my script:
(Get-Content myFile.txt) -replace '(?<pre>"version"[\s]*:[\s]*)(?<V>"[^\"]*")', "`$1`"$Env:BUILD_VERSION`"" | Out-File myFile.txt
Since you are patching a JSON file, regex isn't the way to go. Instead you should parse the JSON, access and change the property you want and write it back:
$filePath = 'your_Path_To_project.json'
$json = (Get-Content $filePath -raw | ConvertFrom-Json)
$json.version = $Env:BUILD_VERSION
$json | ConvertTo-Json -Depth 10 | Set-Content $filePath