I have a pscustomobject I created:
$data = [pscustomobject]#{
CatWeeklyFiles = #()
}
I used a custom function to generate dates for items I am appending to it.
The following is the appending code I run in a for loop.
#Append to the data object
$data.CatWeeklyFiles += #{
FileName = $item.ToString()
Exists = 1
Date = $CatDate.ToString("MM-dd-yyyy")
}
I export this at the end as JSON and this is an example of the first few lines.
How can I sort this in powershell before exporting to JSON so that I can have the date ascending or descending? (Note: it already is in order ascending but I need to know how to change it).
{
"CatWeeklyFiles": [
{
"Date": "04-13-2015",
"FileName": "Week1615CatUpdate.exe",
"Exists": 1
},
{
"Date": "04-20-2015",
"FileName": "Week1715CatUpdate.exe",
"Exists": 1
},
{
"Date": "04-27-2015",
"FileName": "Week1815CatUpdate.exe",
"Exists": 1
},
{
"Date": "05-04-2015",
"FileName": "Week1915CatUpdate.exe",
"Exists": 1
}
]
}
Edit - Adding full code
####Setup
$curDir = Get-Location
$filename = "files.txt"
$pathtocheck = "C:\path\In\Cat_Weekly"
#Create array from the files.txt file
$array = Get-Content $filename
##Function to get date from week/year
Function FirstDateOfWeek
{
param([int]$year, [int]$weekOfYear)
$jan1 = [DateTime]"$year-01-01"
$daysOffset = ([DayOfWeek]::Thursday - $jan1.DayOfWeek)
$firstThursday = $jan1.AddDays($daysOffset)
$calendar = ([CultureInfo]::CurrentCulture).Calendar;
$firstWeek = $calendar.GetWeekOfYear($firstThursday, [System.Globalization.CalendarWeekRule]::FirstFourDayWeek, [DayOfWeek]::Monday)
$weekNum = $weekOfYear
if($firstweek -le 1) { $weekNum -= 1 }
$result = $firstThursday.AddDays($weekNum * 7)
return $result.AddDays(-3)
}
##Create data structure for making JSON
$data = [pscustomobject]#{
CatWeeklyFiles = #()
}
#TEST
#Main Loop
for($i=0; $i -lt $array.Length; $i++){
$item = $array[$i]
#This is for cat weekly files that DO exist.
if((Test-Path -Path $pathtocheck\$item -PathType Leaf) -eq $True){
write-host $i " Update: " $item "exists"
#Extract Week and Year Values
$extWeek = $item.Substring(4,2)
$extYear = "20"+$item.Substring(6,2)
#This handles a bug where the year appears as '99 when it should be '19
if ($extYear -eq 2099){
$extYear = 2019
}
#Run Function
$CatDate= FirstDateOfWeek -year $extYear -weekOfYear $extWeek
#Append to the data object
$data.CatWeeklyFiles += #{
FileName = $item.ToString()
Exists = 1
Date = $CatDate.ToString("MM-dd-yyyy")
}
}
#This is for cat weekly files that DON'T exist.
elseif((Test-Path -Path $pathtocheck\$item -PathType Leaf) -eq $False){
write-host $i " Update: " $item "is missing!"
#Extract Week and Year Values
$extWeek = $item.Substring(4,2)
$extYear = "20"+$item.Substring(6,2)
#This handles a bug where the year appears as '99 when it should be '19
if ($extYear -eq 2099){
$extYear = 2019
}
#Run Function
$CatDate= FirstDateOfWeek -year $extYear -weekOfYear $extWeek
#Append to the data object
$data.CatWeeklyFiles += #{
FileName = $item.ToString()
Exists = 0
Date = $CatDate.ToString("MM-dd-yyyy")
}
}
}
#Output data object as json
$json = $data | ConvertTo-Json | Out-File $curDir\tracking.json
Our comment talk has gotten a little confusing I guess. ;-) Here to show what I meant ... If you have unsorted data input like this:
$InputData = #'
{
"CatWeeklyFiles": [
{
"Date": "2015-04-27",
"FileName": "Week1815CatUpdate.exe",
"Exists": 1
},
{
"Date": "2015-05-04",
"FileName": "Week1915CatUpdate.exe",
"Exists": 1
},
{
"Date": "2015-04-13",
"FileName": "Week1615CatUpdate.exe",
"Exists": 1
},
{
"Date": "2015-04-20",
"FileName": "Week1715CatUpdate.exe",
"Exists": 1
}
]
}
'# |
ConvertFrom-Json
And you output it like this ...
$InputData.CatWeeklyFiles
it looks like this:
Date FileName Exists
---- -------- ------
2015-04-27 Week1815CatUpdate.exe 1
2015-05-04 Week1915CatUpdate.exe 1
2015-04-13 Week1615CatUpdate.exe 1
2015-04-20 Week1715CatUpdate.exe 1
Now you can use Sort-Object to sort it the way you want and safe it to a new vairable...
$NewData =
[pscustomobject]#{
CatWeeklyFiles =
$InputData.CatWeeklyFiles |
Sort-Object -Property Date -Descending
}
When you output it now
$NewData.CatWeeklyFiles
It looks like this
Date FileName Exists
---- -------- ------
2015-05-04 Week1915CatUpdate.exe 1
2015-04-27 Week1815CatUpdate.exe 1
2015-04-20 Week1715CatUpdate.exe 1
2015-04-13 Week1615CatUpdate.exe 1
Now you can convert it to JSON and export it
$NewData |
ConvertTo-Json |
Out-File $curDir\tracking.json
And all of that just because we choose the proper format for the date. ;-)
Related
I have a json block containing keys that have a similar name, each is numbered. I want to iterate over those keys. How can this be achieved?
Eg
$json = #"
{
"output": [
{
"AIeventCheck1": "A",
"AIeventCheck2": "B",
"AIeventCheck3": "C"
}
]
}
"#
$config = $json | ConvertFrom-Json
ForEach ($AIeventCheck in $config.output) {
Write-host AIeventCheck value: $AIeventCheck
}
target output:
A
B
C
Use the psobject memberset to access the individual properties of the object(s):
foreach($AIeventCheck in $config.output){
$AIEventCheckValues = $AIEventCheck.psobject.Properties |Where Name -like 'AIeventCheck*' |ForEach-Object Value
Write-Host AIeventCheck value: $AIeventCheckValues
}
This is what I am doing:
$appParametersXml = [Xml] (Get-Content "$appParameterFilePath\$appParameterFile")
$parameterJsonFile = "$appParameterFilePath\$applicationName"+ "." + $jsonFileName
# Transform the "Parameter" elements into a nested hashtable.
# Convert any values that can be interpreted as [int] to [int] and strip out any comments in the xml file.
$hash = [ordered] #{}
$appParametersXml.Application.Parameters.ChildNodes | Where-Object {$_.NodeType -ne 'Comment'} | % {
$hash[$_.Name] = #{ value = if ($num = $_.Value -as [int]) { $num } else { $_.Value }
}
}
# Wrap the hashtable in a top-level hashtable and convert to JSON.
[ordered] #{
'$schema' = 'https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#'
contentVersion ='1.0.0.0'
parameters = $hash
} | ConvertTo-Json |Out-File $parameterJsonFile
Write-Host "The JSON File is: " $parameterJsonFile
After I build the hash table with existing information from the XML file, I need to add additional parameter values like this Before converting to JSON
"parameters": {
"applicationName": {
"value": "somevalue"
},
"applicationTypeName": {
"value": "somevalue"
},
"applicationTypeVersion": {
"value": "somevalue"
},
Everything that I have tried so far has given me this as additional values. The regular XML values are being converted the correct way but the additional items that I am adding before converting are coming up like this!
"applicationName": "somevalue"
How can i seperate that out on different lines?
So, assuming your input xml file looks something like this ...
<application>
<parameters>
<applicationName>My Awesome App</applicationName>
<!--my awesome comment-->
<applicationTypeName>Mobile App</applicationTypeName>
<applicationTypeVersion>299</applicationTypeVersion>
<!--my other awesome comment-->
</parameters>
</application>
Here is my revised PowerShell ... you can't use if ($num = $_.Value -as [int]) casting as it won't work for 0, as it would be interpreted as false. I prefer to break the steps down and test and check each. Also I've used InnerText for the node value instead of Value as typically Value is evaluated as $null and I'm not sure what your xml looks like.
$fileXml = "./config.xml"
$fileJson = "./config.json"
$xmlContent = [Xml](Get-Content $fileXml)
# Transform the "Parameter" elements into a nested hashtable.
# Set any string values which are integers as [int] and strip out any comments in the xml file.
$parameters = [ordered]#{}
$nodes = $xmlContent.application.parameters.ChildNodes.Where{ $_.NodeType -ne 'Comment' }
foreach ($node in $nodes) {
$parameter = $node.Name
$value = $node.InnerText
if ($value -match "^\d+$") { $value = [int] $value }
$parameters.Add($parameter, #{ value = $value })
}
# if you need to add additional attributes, it's as simple as:
$parameters.Add("newParameter1", #{ value = "newValue1" })
$parameters.Add("newParameter2", #{ value = "newValue2" })
# Wrap the hashtable in a top-level hashtable and convert to JSON.
[ordered]#{
'$schema' = 'https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#'
contentVersion = '1.0.0.0'
parameters = $parameters
} | ConvertTo-Json | Out-File $fileJson
And here is the output saved to the json file:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"applicationName": {
"value": "My Awesome App"
},
"applicationTypeName": {
"value": "Mobile App"
},
"applicationTypeVersion": {
"value": 299
},
"newParameter1": {
"value": "newValue1"
},
"newParameter2": {
"value": "newValue2"
}
}
}
In case anyone else runs into this, it was as simple as doing this after the hash table gets created with the existing XML file
$appParametersFileName = "$appParameterFilePath\$appParameterFile"
$appParametersXml = [Xml] (Get-Content "$appParametersFileName")
$parameterJsonFile = "$appParameterFilePath\$applicationName"+ "." + $jsonFileName
# Transform the "Parameter" elements into a nested hashtable.
# Convert any values that can be interpreted as [int] to [int] and strip out any comments in the xml file.
$hash = [ordered] #{}
$appParametersXml.Application.Parameters.ChildNodes | Where-Object {$_.NodeType -ne 'Comment'} | % {
$hash[$_.Name] = #{ value = if ($num = $_.Value -as [int]) { $num } else { $_.Value }
}
}
$hash["newvalue1"]=#{ value="value1"}
$hash["newvalue2"]=#{ value="value2"}
# Wrap the hashtable in a top-level hashtable and convert to JSON.
[ordered] #{
'$schema' = 'https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#'
contentVersion ='1.0.0.0'
parameters = $hash
} | ConvertTo-Json |Out-File $parameterJsonFile
Write-Host "The JSON File is: " $parameterJsonFile
I try to create a script that will import data given in a csv file to an API, in order to do this, each line of the CSV has to be converted to Json and then posted to the API.
My base data in the csv looks like this:
Users.Name,Users.Mail,Users.AdditionalInfo.String1,Users.AdditionalInfo.Int1
System.String,System.String,System.String,System.Int32
MyName,my#name.com,Hello,1
YourName,your#name.com,GoodBye,2
The first line contains the Json Information, where "Users" is the API resource the data is written to. "Users" can be any other object accessible via the API also, see my comment below.
The second line is to identify the type of the value. This is for later use and not used right now, beginning with line three is data.
The "." is used to specify the "level" of nesting, so the first "." separates the API Object from the first level in the json, the second "." the first level from the second level.
I want a Json to be returned like this:
{
"Name": "MyName",
"Mail": "my#mail.com",
"AdditionalInfo": {
"String1": "Hello",
"Int1": 1
}
}
I use the following code:
$objects = Import-Csv "test.csv" -Encoding UTF8
$validObjects = $objects | Select-Object -Skip 1
$validObjects
ForEach ($object in $validObjects) {
#$apiObject = ($object.psobject.properties.name).Split(".")[0]
$jsonObject = #{}
ForEach ($property in $object.psobject.properties) {
switch($property.Name.Split(".").Count - 1) {
1 {
$jsonObject.Add($property.Name.Split(".")[1], $property.Value)
}
2 {
$tempJsonObject = #{$property.Name.Split(".")[2] = $property.Value}
$jsonObject.Add($property.Name.Split(".")[1], $tempJsonObject)
}
}
}
$jsonObject | ConvertTo-Json
##($object.psobject.properties).Count
}
The problem i face now is that since i have two colums starting with "Users.AdditionalInfo" ist will run into an error, because you can add "AdditionalInfo" only as a key once. Is there an easy way around it?
Seems pretty crazy to have it set up this way. Surely XML would be a better format for the needs. That said, here is what I came up with.
Setting up sample file so others can try this themselves.
$tempfile = New-TemporaryFile
#'
Users.Name,Users.Mail,Users.AdditionalInfo.String1,Users.AdditionalInfo.Int1
System.String,System.String,System.String,System.Int32
MyName,my#name.com,Hello,1
YourName,your#name.com,GoodBye,2
'# | Set-Content $tempfile -Encoding utf8
$csvdata = Import-Csv $tempfile | select -skip 1
Now here's my crazy script
$csvdata | foreach {
$ht = [ordered]#{}
,#($_.psobject.properties) | foreach {
$subprops,$props = $_.where({$_.name -match 'additionalinfo'},'split')
$props | foreach {$ht.Add($_.name,$_.value)}
,#($subprops) | foreach {
$ht.Add("AdditionalInfo",(#{
($_[0].name.split(".")[2]) = $_[0].value
($_[1].name.split(".")[2]) = $_[1].value
}))
}
}
$ht
} | ConvertTo-Json -OutVariable jsonresults
And the output shown plus stored in $jsonresults
[
{
"Users.Name": "MyName",
"Users.Mail": "my#name.com",
"AdditionalInfo": {
"String1": "Hello",
"Int1": "1"
}
},
{
"Users.Name": "YourName",
"Users.Mail": "your#name.com",
"AdditionalInfo": {
"String1": "GoodBye",
"Int1": "2"
}
}
]
A trick I'm sure I picked up from mklement or Mathias that was used twice is
,#(some objects that normally get passed one by one) | foreach
When you add the comma and the array construct, it passes all elements as one vs one at at time. Very helpful in certain times, like these for sure.
The other trick I wanted to highlight is this line
$subprops,$props = $_.where({$_.name -match 'additionalinfo'},'split')
The where method has several modes, one being split. Any that matched additional went to the first variable, the rest went to the second. Hopefully this helps you complete your project.
Edit
Since the items like additionalinfo can be different with varying amount of subproperties, here is a version that will accommodate.
$csvdata | foreach {
$ht = [ordered]#{}
,#($_.psobject.properties) | foreach {
$subprops,$props = $_.where({($_.name.split("."))[2]},'split')
$props | foreach {$ht.Add($_.name,$_.value)}
$subs = $subprops | foreach {$_.name.split(".")[1]} | Select -Unique
foreach($name in $subs)
{
,#($subprops | where name -match $name) | foreach {
$oht = [ordered]#{}
$_ | foreach {$oht[$_.name.split(".")[2]] = $_.value}
$ht.Add($name,$oht)
}
}
}
$ht
} | ConvertTo-Json -OutVariable jsonresults -Depth 5
First data set output
[
{
"Users.Name": "MyName",
"Users.Mail": "my#name.com",
"AdditionalInfo": {
"String1": "Hello",
"Int1": "1"
}
},
{
"Users.Name": "YourName",
"Users.Mail": "your#name.com",
"AdditionalInfo": {
"String1": "GoodBye",
"Int1": "2"
}
}
]
#'
Users.Name,Users.Mail,Users.AnotherPossibility.String1,Users.AnotherPossibility.Int1,Users.AnotherPossibility.Int2,Users.AdditionalInfo.String1,Users.AdditionalInfo.Int1
System.String,System.String,System.String,System.Int32,System.String,System.Int32
MyName,my#name.com,Hello,1,3,Bonjour,5
YourName,your#name.com,GoodBye,2,4,Adios,6
'# | Set-Content $tempfile -Encoding utf8
Second data set output
[
{
"Users.Name": "MyName",
"Users.Mail": "my#name.com",
"AnotherPossibility": {
"String1": "Hello",
"Int1": "1",
"Int2": "3"
},
"AdditionalInfo": {
"String1": "Bonjour",
"Int1": "5"
}
},
{
"Users.Name": "YourName",
"Users.Mail": "your#name.com",
"AnotherPossibility": {
"String1": "GoodBye",
"Int1": "2",
"Int2": "4"
},
"AdditionalInfo": {
"String1": "Adios",
"Int1": "6"
}
}
]
I need to show id_products as a result, the remaining data I have a photo id and url pictures.
Below is the code which I extract data from
$files_ro = "products.csv"
$Ident = Import-Csv -Path $files_ro -Header id_product | select-object -skip 1
foreach ($idka in $ident)
{
$idp = $idka.id_product
$request_n = "http://api.url/"+ $idp +""
foreach($d1 in $request_n)
{
Invoke-WebRequest $d1 |
ConvertFrom-Json |
Select-Object -Expand data |
Select -expand extended_info |
select -expand images |
Select id,url
}
}
files
- product.csv
"id_product"
"21221"
"23526"
"23525"
"24074"
"21302"
"24372"
"21272"
"21783"
"27268"
"21776"
json
{
data: {
id: 21221,
extended_info: {
images: [
{
id: 34380,
url: photos1.jpg
},
{
id: 34381,
url: photos2.jpg
},
{
id: 34382,
url: photos3.jpg
}
],
}
}
}
I would like it to look like this:
id_product,id(images), url
21221,34380,photos1.jpg
21221,34381,photos2.jpg
21221,34382,photos3.jpg
You can help me somehow ?
Your provided JSON is not valid. However, I would use a PSCustomObject to create the desired result:
$json = #'
{
"data": {
"id": 21221,
"extended_info": {
"images": [{
"id": 34380,
"url": "photos1.jpg"
}, {
"id": 34381,
"url": "photos2.jpg"
}, {
"id": 34382,
"url": "photos3.jpg"
}
]
}
}
}
'# | ConvertFrom-Json
$json.data.extended_info.images | ForEach-Object {
[PSCustomObject]#{
id_product = $json.data.id
"id(images)" = $_.id
url = $_.url
}
}
Output:
id_product id(images) url
---------- ---------- ---
21221 34380 photos1.jpg
21221 34381 photos2.jpg
21221 34382 photos3.jpg
To convert the result to CSV, just add | ConvertTo-Csv -NoTypeInformation after the last curly bracket to get the following output:
"id_product","id(images)","url"
"21221","34380","photos1.jpg"
"21221","34381","photos2.jpg"
"21221","34382","photos3.jpg"
As your product.csv already has the same header it isn't necessary to supply one and then skip the first line.
Your single URL $request_n also doesn't need to be iterated with a foreach
I suggest to store the result from the webrequest and convertedfrom-json into var $Json and proceed with Martin Brandls good answer.
## Q:\Test\2019\05\24\SO_56287843.ps1
$files_ro = "products.csv"
$Ident = Import-Csv -Path $files_ro
$Data = foreach ($idka in $ident){
$request_n = "http://api.url/{0}" -f $idka.id_product
$Json = Invoke-WebRequest $request_n | ConvertFrom-Json
# inserted code from Martin Brandl's good answer
$Json.data.extended_info.images | ForEach-Object {
[PSCustomObject]#{
id_product = $json.data.id
"id(images)" = $_.id
url = $_.url
}
}
}
$Data
$Data | Export-Csv ProductImages.csv -NoTypeInformation
#$Data | Out-Gridview
I'm trying to automate some data pipelines with Powershell, but I'm kinda stuck with converting a JSON list to a single cell per row in a CSV file. Hope some of you can help me out.
The JSON I get looks like the following:
{"result": [
{
"uid": "1",
"EducationHistory": []
},
{
"uid": "2",
"EducationHistory": []
},
{
"uid": "3",
"EducationHistory": []
},
{
"uid": "4",
"EducationHistory": {
"10466632": {
"euid": 10466632,
"degree": "Highschool",
"educationLevel": null
},
"10466634": {
"euid": 10466634,
"degree": "Law",
"educationLevel": "batchelor"
},
"10466635": {
"euid": 10466635,
"degree": "Law",
"educationLevel": "master"
}
}
},
{
"uid": "5",
"EducationHistory": {
"10482462": {
"euid": 10482462,
"degree": "IT",
"educationLevel": "master"
}
}
}
]
}
What I want to do is collect the educationLevels per uid in one column. So something like this:
uid | educationLevel
----+------------------
1 |
2 |
3 |
4 | barchelor, master
5 | master
Normally I would like Expandproperty to get down to a lower level, but this doesn't work for this case because every EducationHistory entry is behind a euid for that specific entry. Expanding every single one of them like in the example below isn't workable because of the number of records.
So I think I need something of a loop, but I don't know how. Hope you can help me. First post here and a Powershell newbie, so I hope my question is clear. Please let me know if you need more info.
The code for one entry, as example:
$json = Get-content -raw -path C:\TEMP\File.json
(ConvertFrom-Json -InputObject $json).result |
Select-Object uid,
#Expand one of the entries:
#{Name = "Edu.Level";E={$_.EducationHistory | Select-Object -
expandproperty 10466632 |Select-Object -expandpropert degree }} |
Format-Table
$content = Get-Content .\test.json
$result = ($content | ConvertFrom-Json).result
$totalResult = #()
foreach($res in $result) {
$tempArray = #()
if($res.EducationHistory -ne $null) {
$properties = $res.EducationHistory | Get-Member -MemberType NoteProperty
foreach($property in $properties) {
$eduLevel = $res.EducationHistory.$($property.Name).educationLevel
if(![String]::IsNullOrEmpty($eduLevel)) {
$tempArray += $eduLevel
}
}
}
$totalResult += [PSCustomObject]#{
uid = $res.uid
educationLevel = $tempArray -join ", "
}
}
$totalResult
This will output desired result for the input you have provided.
The trickiest part is the value of EducationHistory property. You have to use Get-Member cmdlet (see Get-Help Get-Member) to get the properties of the current object in loop. Then using the name of the property to access the educationLevel.
Your first question, my first answer I believe :) Similar to the last answer. You need to jump through the hoop of finding the object names in EducationalHistory to reference them.
$json = (Get-content C:\TEMP\File.json | ConvertFrom-Json).result
$results = #()
foreach ( $u in $json)
{
foreach ( $h in $u.EducationHistory)
{
$results += $h.PSObject.properties.Name | ForEach-Object{new-object PSObject -property #{ uid=$u.uid; degree=$h.$_.degree}}
}
}
$results | ConvertTo-Csv | select -skip 1