Get values from json file in string with separators - json

I have a json file with following format:
{
"total_count": 57,
"incomplete_results": false,
"items": [
{
"id": 123456,
"node_id": "dfghdfjfgjftgjhtfyjh",
"name": "Firstrepo",
"full_name": "MyOrganization/Firstrepo",
[SKIP]
},
{
"id": 4756758,
"node_id": "dtghtfjgjyuj",
"name": "Secondrepo",
"full_name": "MyOrganization/Secondrepo",
[SKIP]
},
{
"id": 568578567,
"node_id": "dsgdfghftghtfyjtyj",
"name": "Anotherrepo",
"full_name": "MyOrganization/Anotherrepo",
[SKIP]
},
{
"id": 58567856,
"node_id": "sjdhfbgsdjfgjsdfjjs",
"name": "Somerepo",
"full_name": "MyOrganization/Somerepo",
[SKIP]
},
How can I get values of repositories and write them in variable or file in csv format. Like:
Firstrepo,Secondrepo,Anotherrepo,Somerepo
Script to get values:
CONFIG Get-list of repos from Github
$GithubAPIURL = "https://api.github.com"
$Suffix = "/search/repositories"
$Additions = "?q=org:MYORG:SEARCH&page=1&per_page=100"
$GithubAPIToken = "&access_token=MYTOKEN"
PROCESS
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$variable = ((Invoke-Webrequest $GithubAPIURL$Suffix$Additions$GithubAPIToken).content | ConvertFrom-Json)
$jsonfile = "c:\a\test.json"
$variable | ConvertTo-Json | Out-File $jsonfile -Encoding UTF8
Get-List of names
$Reponames = (Get-Content $jsonfile -Encoding UTF8 | ConvertFrom-Json)

To get the Names of the Repository use:
(Get-Content $jsonfile -Encoding UTF8 | ConvertFrom-Json).items.name
To convert it to a .csv file:
(Get-Content $jsonfile -Encoding UTF8 | ConvertFrom-Json).items | ConvertTo-Csv -Delimiter ';' -NoTypeInformation | Out-File $csvFilePath

Related

Powershell: JSON to Excel

I have had a problem for days and am now reporting here. I want to export several JSON files to an Excel spreadsheet. The JSON keys should form the headers and the values should be listed under the headers. Unfortunately, I have zero understanding of Powershell and can't get any further with the help of other threads, as I also don't understand why something works or doesn't work the way it does.
The json files look something like this
{"dataCollection": [
{
"objectID": 000001,
"randomID": 123,
"desc": "The sky is blue",
"startTime": "2022-03-15T11:31:56.510",
"endTime": "2022-03-15T11:31:56.511",
"caseOne": true,
"caseTwo": false,
"caseThree": null
},
{
"objectID": 333222,
"randomID": 456,
"desc": "example",
"startTime": "2022-03-15T11:31:56.510",
"endTime": "2022-03-15T11:31:56.511",
"caseOne": false,
"caseTwo": true,
"caseThree": null
},
{
"objectID": 111111,
"randomID": 789,
"desc": "Mo-Fr 60% 20-24",
"startTime": "2022-03-15T11:31:56.510",
"endTime": "2022-03-15T11:31:56.511",
"caseOne": false,
"caseTwo": false,
"caseThree": null
}
]}
My current code looks like this
$contentJson = Get-Content -Raw -Path $jsonInput | ConvertFrom-Json
$obj_list = $contentJson | Select-Object #{Name='Name';Expression={$_}}
$obj_list | Export-Csv $csvOutput -NoType -Delimiter "`t" -Encoding Unicode
(Get-Content -Path $csvOutput -Raw).replace('"','') | Set-Content -Path $csvOutput
This does give me a CSV with the information from the json, however it is transferred cell by cell and I have no idea how to create headers. Further this works at all only, as soon as I remove in the first line of the JSON (in this case {"DataCollection":), otherwise in the Excel table only the following is written: #{ttDebugTage=System.Object[]}
My goal is something looking like this:
Excel:
This is the first time I'm working with Powershell and unfortunately I'm completely lacking in understanding, so I would appreciate any help.
$contentJson = #'
{"dataCollection": [
{
"objectID": 000001,
"randomID": 123,
"desc": "The sky is blue",
"startTime": "2022-03-15T11:31:56.510",
"endTime": "2022-03-15T11:31:56.511",
"caseOne": true,
"caseTwo": false,
"caseThree": null
},
{
"objectID": 333222,
"randomID": 456,
"desc": "example",
"startTime": "2022-03-15T11:31:56.510",
"endTime": "2022-03-15T11:31:56.511",
"caseOne": false,
"caseTwo": true,
"caseThree": null
},
{
"objectID": 111111,
"randomID": 789,
"desc": "Mo-Fr 60% 20-24",
"startTime": "2022-03-15T11:31:56.510",
"endTime": "2022-03-15T11:31:56.511",
"caseOne": false,
"caseTwo": false,
"caseThree": null
}
]}
'#
($contentJson | ConvertFrom-Json).dataCollection |
Select-Object -Property objectID, randomID, desc |ConvertTo-Csv -Delimiter "`t"
"objectID" "randomID" "desc"
"1" "123" "The sky is blue"
"333222" "456" "example"
"111111" "789" "Mo-Fr 60% 20-24"
################## PFADE ###################
$jsonDirectory = 'DIR\TO\JSON\FILES'
$csvFile = 'DIR\TO\OUTPUT\FILE.CSV'
################ Variablen #################
$excel = New-Object -ComObject Excel.Application
############################################
#ALLE JSON FILES
$jsonFiles = Get-ChildItem -Path $jsonDirectory -Filter *.json
#FILTER
$unwantedKeys = #("Example1", "Example2", "Example3")
#ARRAY ZWISCHENSPEICHER
$jsonData = #()
foreach ($jsonFile in $jsonFiles) {
#Lädt JSON File
$json = Get-Content $jsonFile.FullName | ConvertFrom-Json
$firstKey = ($json.PSObject.Properties.Name)
#Filter
$json = $json.$firstKey | Select-Object * -ExcludeProperty $unwantedKeys
#Content ins Array
$jsonData += $json
}
#Erstellt CSV und Importiert JSON Content
$jsonData | Export-Csv $csvFile -NoTypeInformation -Delimiter "`t" -Encoding Unicode
#Anpassen von Spaltenbreite auf Valuelänge
$Workbook = $excel.Workbooks.Open($csvFile)
$Worksheet = $Workbook.Sheets.Item(1)
$range = $worksheet.UsedRange
$range.EntireColumn.AutoFit()
$excel.Visible = $True

Seperating Json objects based on keyvalue

currently i am working on fetching Azure ad application expiry status on that i have pulled around 1700 ad applications and i have added one key pair(status) to the json object based on the secret expiry date
1) valid 2) Expired 3) expiring soon
so i have extracted all applications to a json file now i need to split single file into 3 files based on status as mentioned below
[
{
"DisplayName": "Reporter-dev",
"ObjectId": null,
"ApplicationId": {
"value": "62838283828288282828828288282828",
"Guid": "62838283828288282828828288282828"
},
"KeyId": "62838283828288282828828288282828",
"Type": "Password",
"StartDate": {
"value": "/Date(1590537256000)/",
"DateTime": "27 May 2020 05:24:16"
},
"EndDate": {
"value": "/Date(1653609256000)/",
"DateTime": "27 May 2022 05:24:16"
},
"Ownername": "shetty#gmail.com",
"Status": "Valid"
},
{
"DisplayName": "azure-cli-2018",
"ObjectId": null,
"ApplicationId": {
"value": "52388282828828288273673282932739223",
"Guid": "52388282828828288273673282932739223"
},
"KeyId": "52388282828828288273673282932739223",
"Type": "Password",
"StartDate": {
"value": "/Date(1568849784000)/",
"DateTime": "19 September 2019 05:06:24"
},
"EndDate": {
"value": "/Date(1600472184000)/",
"DateTime": "19 September 2020 05:06:24"
},
"Ownername": "joseph#gmail.com",
"Status": "Expired"
},
{
"DisplayName": "azure-cli-2019",
"ObjectId": null,
"ApplicationId": {
"value": "26382882828828282882828282828",
"Guid": "26382882828828282882828282828"
},
"KeyId": "26382882828828282882828282828",
"Type": "Password",
"StartDate": {
"value": "/Date(1576143476000)/",
"DateTime": "12 December 2019 15:07:56"
},
"EndDate": {
"value": "/Date(1607765876000)/",
"DateTime": "12 December 2020 15:07:56"
},
"Ownername": "zzzzzzzzz#gmail.com",
"Status": "About to Expire"
}
]
The below will split out the JSON based on the status and convert the data back to JSON. Change $JSONPath, $ValidPath, $ExpiredPath, $ExpiredSoonPath to the paths you require, the ones currently populated are what I have used for testing.
The contents of $JSONPath must have valid JSON to be able to work, whilst this is probably not the most efficient nor elegant it should do what you need.
$JSONPath = "C:\PS\JT\JSON.txt"
$JSONObj = Get-Content $JSONPath | ConvertFrom-Json
$ValidPath = "C:\PS\JT\Valid.txt"
$ExpiredPath = "C:\PS\JT\Expired.txt"
$ExpireSoonPath = "C:\PS\JT\ExpireSoon.txt"
$JSONObj | Where {$_.Status -eq "Valid"} | ConvertTo-Json | Out-File $ValidPath
$JSONObj | Where {$_.Status -eq "Expired"} | ConvertTo-Json | Out-File $ExpiredPath
$JSONObj | Where {$_.Status -eq "About to Expire"} | ConvertTo-Json | Out-File $ExpireSoonPath
Something very simple like the below example should work in theory but since you won't show your code there's a bunch of guess work here...
$collection = $json | ConvertFrom-Json
$valid = #(); $exp = #(); $soon = #(); $unknown = #()
foreach($item in $collection) {
switch ($item.Status)
{
'Valid' { $valid += $item }
'Expired' { $exp += $item }
'About to Expire' { $soon += $item }
Default { $unknown += $item}
}
}
$valid | Out-File .\Valid.txt -Append
$exp | Out-File .\Expired.txt -Append
$soon | Out-File .\AboutToExpire.txt -Append
$unknown | Out-File .\Unknown.txt -Append
Or if json is the desired output as in the other example;
$valid | ConvertTo-Json | Out-File .\Valid.txt -Append
etc, etc
Adding to the other answers, here is another solutions using pipelines.
Get-Content -Path .\sample.json -Raw | ConvertFrom-Json |
ForEach-Object { $_ | ConvertTo-Json | Out-File -FilePath "$($_.Status).json" }
Explanation
Parse JSON content using Get-Content, also making sure we use the -Raw switch to return the contents as one string with the newlines preserved. We don't need the contents as an array of strings(default behaviour).
Pipe to ConvertFrom-Json to deserialize the JSON into a System.Management.Automation.PSCustomObject.
Iterate over the objects using Foreach-Object and convert the object to a JSON file using ConvertTo-Json and Out-File. Note the default depth for ConvertTo-Json is 2, so if your JSON files end up having deeper levels, then you will need to specify a larger depth using the -Depth switch.
If its easier understand, you can also use a regular foreach loop here:
$json = Get-Content -Path .\sample.json -Raw | ConvertFrom-Json
foreach ($object in $json) {
$object | ConvertTo-Json | Out-File -FilePath "$($object.Status).json"
}

Trying to convert CSV into a particular JSON format

I need to convert a CSV to a particular JSON format but having trouble.
I currently have created the below powershell code which takes a CSV file with multiple columns and data for each column
enter code here $csvcontent = get-content "C:\tmp\vmfile.csv" | select -Skip 1
$Json =foreach($line in $csvcontent){
$obj = [PSCustomObject]#{
description = ($line -split ",")[0] -replace "`""
requestedFor = ($line -split ",")[1] -replace "`""
VMs = #{
vmType = $(($line -split ",")[5] -replace "`"");
environment = $(($line -split ",")[6] -replace "`"");
vmdescription = $(($line -split ",")[7] -replace "`"");
function = $(($line -split ",")[8] -replace "`"");
datacenter = $(($line -split ",")[9] -replace "`"");
Size = $(($line -split ",")[10] -replace "`"");
adDomain = $(($line -split ",")[11] -replace "`"");
Hostname = $(($line -split ",")[12] -replace "`"")
}
ExtraDisks = #{
VolumeName = $(($line -split ",")[14] -replace "`"");
VolumeLetter = $(($line -split ",")[15] -replace "`"");
Size = $(($line -split ",")[16] -replace "`"")
}
}
$obj | ConvertTo-Json
}
$json -replace '(?<=:\s+){','[ {' -replace '(?<="\s+)}','} ]'
This then generates the following json file which is not what i need as i want it all to come under the VM brackets not have a separate one for each VM
enter code here
{
"requestedFor": "John Doe",
"VMs": {
"Size": "Medium",
"datacenter": "DC1",
"environment": "dev",
"adDomain": "mydomain.com",
"vmType": "Windows Server",
"vmdescription": "VM Build1",
"function": "app",
"Hostname": "VMBuild1"
},
"ExtraDisks": {
"VolumeLetter": "G",
"Size": "10",
"VolumeName": "Logs"
}
}
{
"requestedFor": "John Doe",
"VMs": {
"Size": "Medium",
"datacenter": "DC2",
"environment": "prod",
"adDomain": "mydomain.com",
"vmType": "Windows Server",
"vmdescription": "VM Build2",
"function": "app",
"Hostname": "VMBuild2"
},
"ExtraDisks": {
"VolumeLetter": "E",
"Size": "50",
"VolumeName": "Data"
}
}
but what i need it to look like this
enter code here
{
"requestedFor": "John Doe",
"VMs": [ {
"vmType": "Windows Server",
"environment": "dev",
"description": "VMBuild1",
"function": "app",
"datacenter": "DC1",
"size": "Medium",
"adDomain": "mydomain.com",
"Hostname": "VMBuild1",
"ExtraDisks": [ {
"VolumeName": "Logs",
"VolumeLetter": "G",
"VolumeSize": 10
}
]
},
{
"vmType": "Windows Server",
"environment": "prod",
"description": "VMBuild2",
"function": "app",
"datacenter": "DC2",
"size": "Medium",
"adDomain": "mydomain.com",
"Hostname": "VMBuild2",
"ExtraDisks": [ {
"VolumeName": "Data",
"VolumeLetter": "E",
"VolumeSize": 50
}
]
}
]
}
Here is the CSV file contents
vmType environment description function datacenter Size adDomain Hostname VolumeName VolumeLetter VolumeSize
Windows Server dev VMBuild1 app DC1 Medium mydomain.com VMBUILD1 Logs G 10
Windows Server prod VMBuild2 app DC2 Medium mydomain.com VMBUILD2 Data E 50
Although your example CSV doesn't show it (copy/paste from Excel), I'm assuming it looks like this when opened in Notepad:
"vmType","environment","description","function","datacenter","Size","adDomain","Hostname","VolumeName","VolumeLetter","VolumeSize"
"Windows Server","dev","VMBuild1","app","DC1","Medium","mydomain.com","VMBUILD1","Logs","G","10"
"Windows Server","prod","VMBuild2","app","DC2","Medium","mydomain.com","VMBUILD2","Data","E","50"
"Windows Server","dev","VMBuild1","app","DC1","Medium","mydomain.com","VMBUILD1","Scripts","H","25"
The CSV does not have a column for RequestedFor, so the code below uses that as hardcoded variable.
Instead of reading the csv as string array and doing a lot of splitting and removing quote characters, you need to use Import-Csv.
After that, the only thing left to do is the way you want the final JSON formatted.
$requestor = 'John Doe'
$csvData = Import-Csv -Path 'D:\Test\vmfile.csv'
# get an array of PSObjects
# we use 'Group-Object Hostname' here to allow VMs with multiple extra disks
$allVMs = $csvData | Group-Object Hostname | ForEach-Object {
$disks = $_.Group | Select-Object VolumeName, VolumeLetter, VolumeSize
$vm = $_.Group[0] | Select-Object * -ExcludeProperty VolumeName, VolumeLetter, VolumeSize
$vm | Add-Member -MemberType NoteProperty -Name 'ExtraDisks' -Value #($disks)
# output the VM object
$vm
}
# combine the requestor, main element 'VMs' and the objects
# gathered above into a new object and convert that to JSON
[PsCustomObject]#{
RequestedFor = $requestor
VMs = #($allVMs)
} | ConvertTo-Json -Depth 4
Output:
{
"RequestedFor": "John Doe",
"VMs": [
{
"vmType": "Windows Server",
"environment": "dev",
"description": "VMBuild1",
"function": "app",
"datacenter": "DC1",
"Size": "Medium",
"adDomain": "mydomain.com",
"Hostname": "VMBUILD1",
"ExtraDisks": [
{
"VolumeName": "Logs",
"VolumeLetter": "G",
"VolumeSize": "10"
},
{
"VolumeName": "Scripts",
"VolumeLetter": "H",
"VolumeSize": "25"
}
]
},
{
"vmType": "Windows Server",
"environment": "prod",
"description": "VMBuild2",
"function": "app",
"datacenter": "DC2",
"Size": "Medium",
"adDomain": "mydomain.com",
"Hostname": "VMBUILD2",
"ExtraDisks": [
{
"VolumeName": "Data",
"VolumeLetter": "E",
"VolumeSize": "50"
}
]
}
]
}
Of course, you can save this in a json file, by appending | Set-Content -Path 'TheOutputFile.json' to it.
P.S. PowerShell does not produce 'pretty' json. If you need to convert it to properly spaced json, see my function Format-Json
You don't need to parse the csv yourself.
That's what ConvertFrom-Csv / Import-CSV are for.
Here's how I'd do it.
$CSVObj = get-content "C:\tmp\vmfile.csv" -Raw | ConvertFrom-Csv
$CSVObj | ConvertTo-Json | Set-Content "C:\tmp\vmfile.json"
That's all !
But let's go further. There was no CSV sample in your question so one might assume that the output JSON might still be incorrect. How would you make sure to have the format you want ?
By creating a brand new object structure from the imported object and then exporting it.
Here's a simple expression of what that might look like.
$CSVObj = get-content "C:\tmp\vmfile.csv" -Raw | ConvertFrom-Csv
# Create a new object from $csvObj that you will then export to csv
$Output = foreach ($item in $CSVObj) {
[PSCustomObject]#{
Requester = $item.requestedFor
VMs = $item.VMs
Count = $item.VMs.Count
}
}
$output | ConvertTo-Json | Set-Content "C:\tmp\vmfile.json"
You would then have successfully modified the json to be output to fit your needs.

How can I craft a csv from multi level json in powershell?

I am trying to parse a JSON response that looks like
{
"Objects": [
{
"Name": "FirstName",
"Type": "XXXX",
},
{
"Name": "SecondName",
"Type": "YYYY",
},
{
"Name": "ThirdName",
"Type": "ZZZZ",
},
],
"TotalCount": 127
}
I want a CSV formatted like
"Name","Type"
"FirstName","XXXX"
"SecondName","YYYY"
and so on.
I tried creating a PSCustomObject and using Select-Object with it to generate the CSV but it does not give me the desired output.
My code:
$report=$null
foreach($obj in $json){
$item=$obj | ConvertFrom-Json
$report=[pscustomobject]#{
Name=($item.Objects.Name | Out-String).Trim()
Type=($item.Objects.Type | Out-String).Trim()
}
$report | Select-Object Name,Type | Export-Csv "PATH"`
Gives me CSV that looks like:
"Name","Type"
"FirstName
SecondName
ThirdName",
"XXXX
YYYY
ZZZZ"
if you really want the total count in the csv as well, here is one thing you could do:
$psobjarray = ($json | convertfrom-json)
$psobjarray.objects | Select-Object Name,Type | Export-Csv $Path
"Total Count: $($psobjarray.totalcount)" | out-file $Path -append

How to save a JSON object to a file using Powershell?

I have converted the following JSON file to powershell representation object.
{
"computer": [
{
"children": [
{
"children": [ {
"children": [ {
"path": "T:\Dropbox\kvaki.html",
"name": "kvaki",
"type": "url",
"url": "http://example.com"
} ],
"path": "T:\Dropbox\",
"name": "Njusha",
"type": "folder"
}, {
"path": "T:\Dropbox\Europa.html",
"name": "Europa",
"type": "url",
"url": "http://example.com"
}, {
"path": "T:\Dropbox\math.html",
"name": "math",
"type": "url",
"url": "http://example.com"
} ],
"path": "T:\Dropbox\",
"name": "Money",
"type": "folder"
}
],
"full_path_on_file_sys": "T:\Dropbox\"
}
]
}
After doing some computations with powershell representation I would like to save it to file as JSON.
But command $jsonRepresentation | ConvertTo-Json | Out-File "D:\dummy_path\file.json" saves it in this way
{
"computer": [
{
"children": " ",
"full_path_on_file_sys": "T:\Dropbox\"
}
]
}
Question: how to achieve correct saving of complex powershell JSON representation?
-depth argument for ConvertTo-Json solves the issue.
$jsonRepresentation | ConvertTo-Json -depth 100 | Out-File "D:\dummy_path\file.json"
Just pipe it to Set-Content, or Out-File:
Get-Process powershell |
ConvertTo-Json |
Set-Content json.txt
If you want to both view the output and save it to file, you can pipe the tee command.
Get-Process powershell | ConvertTo-Json | Tee-Object json.txt
$json.properties.metadata | ConvertTo-Json -Compress
if you are stuck with PowerShell Version 2, the JSON module of Joel Bennett from the 'PowerShell Code Repository' might help.
1) The below command can be used to convert a json to CSV
Example:
Get-Content package.json | Out-String | ConvertFrom-Json | Select parameter1, parameter2, parameter3 | ConvertTo-Csv -NoTypeInformation | Format-Table >> C:\JenkinsWorkspace\Result.csv
Get-Content: This is like "cat" command in linux which will get all data of file "package.json" and converts from Json (Using ConvertFrom-Json function) extracting the details of only required parameters and then converting them into CSV using "ConvertTo-Csv" function without any unwanted Type Headers and formatting them into Table.
2) The above result can also be formatted into proper csv to view it as Excel format without any duplicates and also having Text-To-Column conversion using below command:
Import-Csv "C:\Result.csv" -delimiter "," | Sort-Object _from -Unique | Export-csv "C:\FINAL_REPORT_$date.csv"