Export fields with multiple nested values from JSON to CSV - json

I need to import some JSON data am getting into my database. I'd like to use PowerShell and figured out a little, but I need help getting the final piece together.
Here is my existing PowerShell script.
Get-Content -Path "t:\1.json" |
ConvertFrom-Json |
Select-Object -expand data |
ConvertTo-Csv -NoTypeInformation |
Set-Content "t:\1.csv"
It does everything but doesnot get the badges details out in single cav along with user name. My question is, how can I get all details in single csv?
Here is my JSON data:
{
"data": [
{
"name": "Shiva",
"email": "Shiva#example.com",
"organization": "Team",
"badges": {
"data": [
{
"name": "AWS Certified Developer",
"id": "001"
}
]
},
"id": "1001"
},
{
"name": "Rudra",
"email": "Rudra#example.com",
"organization": "Team",
"badges": {
"data": [
{
"name": "Certified Google Associate Cloud Engineer",
"id": "006"
}
]
},
"id": "1002"
},
{
"name": "Alazar",
"email": "Alazar#example.com",
"organization": "Team",
"badges": {
"data": [
{
"name": "Google Cloud Data Engineer Certified",
"id": "007"
},
{
"name": "Google Certified Professional Cloud Network Engineer",
"id": "008"
},
{
"name": "AWS Solution Architect",
"id": "009"
},
{
"name": "Certified Google Associate Cloud Engineer",
"id": "006"
}
]
},
"id": "1003"
}
]
}

You need to add new rows of data for each badge a person can have.
Something like this:
(Get-Content -Path 't:\1.json' -Raw | ConvertFrom-Json).data | ForEach-Object {
foreach ($badge in $_.badges.data) {
$_ | Select-Object *,
#{Name = 'badge_id'; Expression = { $badge.id }},
#{Name = 'badge_name'; Expression = { $badge.name }} -ExcludeProperty badges
}
} | Export-Csv -Path 't:\1.csv' -NoTypeInformation
After this, file 't:\1.csv' will look like this:
"name","email","organization","id","badge_id","badge_name"
"Shiva","Shiva#example.com","Team","1001","001","AWS Certified Developer"
"Rudra","Rudra#example.com","Team","1002","006","Certified Google Associate Cloud Engineer"
"Alazar","Alazar#example.com","Team","1003","007","Google Cloud Data Engineer Certified"
"Alazar","Alazar#example.com","Team","1003","008","Google Certified Professional Cloud Network Engineer"
"Alazar","Alazar#example.com","Team","1003","009","AWS Solution Architect"
"Alazar","Alazar#example.com","Team","1003","006","Certified Google Associate Cloud Engineer"

Related

List inside JSON Powershell

I am trying to parse the following JSON into a PSCustomObject
[
{
"tags": [
{
"tagName": "Microsoft Teams"
},
{
"tagName": "Worldwide (Standard Multi-Tenant)"
},
{
"tagName": "General Availability"
},
{
"tagName": "Web"
},
{
"tagName": "Desktop"
}
],
"tagsContainer": {
"products": [
{
"tagName": "Microsoft Teams"
}
],
"cloudInstances": [
{
"tagName": "Worldwide (Standard Multi-Tenant)"
}
],
"releasePhase": [
{
"tagName": "General Availability"
}
],
"platforms": [
{
"tagName": "Web"
},
{
"tagName": "Desktop"
}
]
},
"id": 51230,
"title": "Microsoft Teams: New file sharing experience",
"description": "Streamline sharing with Microsoft Teams. You can now create a shareable link for any file stored in Teams and directly set the appropriate permissions. Additionally, you can also set permissions for files stored in SharePoint or OneDrive while composing a private chat or starting a channel conversation.",
"status": "Launched",
"moreInfoLink": "https://techcommunity.microsoft.com/t5/microsoft-sharepoint-blog/rich-new-file-and-sharing-experiences-throughout-microsoft-365/ba-p/960129",
"publicRoadmapStatus": "Include this month",
"created": "2019-05-08T07:00:00",
"modified": "2022-01-13T00:05:19.663",
"publicDisclosureAvailabilityDate": "March CY2021",
"publicPreviewDate": ""
}
]
I have that JSON stored in a variable like this:
$RoadmapContent = Get-Content -Raw -Path ".\M365Roadmap_single.json" | ConvertFrom-Json
And this would be my PSCustomObject:
$RoadmapItems =[PSCustomObject]#{
Title = $($RoadmapContent.title)
Tags = $($RoadmapContent.tags)
}
I would like to have this tags in a single column, like this: Microsoft Teams, Worldwide (Standard Multi-Tenant), General Availability, Web, Desktop
I have tried to accomplish that (unsuccessfully) like this:
$RoadmapContent.Tags -join ";"
How can I parse this?
Many thanks!
Edit:
I can access the tags like this:
foreach ($tag in $RoadmapContent.tags) {Write-Host $Tag.tagName}
But cannot figure out how to use this in a PSCustomObject
Figured it out thanks to a comment in the original question, here's the full code:
$RoadmapContent = Get-Content -Raw -Path ".\M365Roadmap_single.json" | ConvertFrom-Json
$RoadmapItems =[PSCustomObject]#{
Title = $($RoadmapContent.title)
Tags = $($RoadmapContent.tags.tagname -join ', ')
}
$RoadmapItems

How to select words after /slash from a json file in a powershell script

I've a json file from a az cli command, I want to deploy with a powershell script from a tool like jenkins or rundeck, I need to take the field "name" from json but not entirely only the word after forward slash
[
{
"Group": "KC-EMEA-RSGP-BPTRAINING-01",
"id": "/subscriptions/KC-EMEA-RSGP-BPTRAINING-01/providers/Microsoft.Sql/servers/",
"name": "kc-emea-sqsrv-bptraining-dev-01/master"
},
{
"Group": "KC-EMEA-RSGP-NAVISIONKM-DEV-01",
"id": "/subscriptions/KC-EMEA-RSGP-NAVISIONKM-DEV-01/providers/Microsoft.Sql/servers/",
"name": "km-emea-sqsrv-navision-tst-01/km-emea-sqdb-navision-tst-01"
},
{
"Group": "KC-EMEA-RSGP-NAVISIONKM-DEV-01",
"id": "/subscriptions/KC-EMEA-RSGP-NAVISIONKM-DEV-01/providers/Microsoft.Sql/servers/",
"name": "km-emea-sqsrv-navision-tst-01/master"
},
{
"Group": "KC-EMEA-RSGP-PROJECTS-DEV-01",
"id": "/subscriptions/KC-EMEA-RSGP-PROJECTS-DEV-01/providers/Microsoft.Sql/servers/",
"name": "kc-emea-sqsrv-projects-dev-01/KC-EMEA-SQDB-BPTRAINING-TRAINEE-01"
}
]
$file = "pathtojsonfile.json"
$jsonContent = Get-Content $file | ConvertFrom-Json;
$namedb = $jsonContent.name;
$value= $namedb.Substring($namedb.IndexOf('/')+1) --> this doesn't work.
Try
$namedb[0].Substring($namedb[0].IndexOf('/')+1)
And for all of them:
foreach($name in $namedb) {$name.Substring($name.IndexOf('/')+1)}

jq - Find a JSON object based on one of its values and get another value from it

I've started using jq just very recently and I would like to know if something like this is even possible.
Example:
{
"name": "device",
"version": "1.0.0",
"address": [
{
"address": "10.1.2.3",
"interface": "wlan1_wifi"
},
{
"address": "10.1.2.5",
"interface": "wlan2_link"
},
{
"address": "10.1.2.4",
"interface": "ether1"
}
],
"wireless": [
{
"name": "wlan1_wifi",
"type": "5Ghz",
"ssid": "wifi"
},
{
"name": "wlan2_link",
"type": "2Ghz",
"ssid": "link"
}
]
}
Firstly let's transform the example to this json object:
cat json | jq '. | {"name": ."name", "version": ."version", "wireless": [."wireless"[] | {"name": ."name", "type": ."type", "ssid": ."ssid"}]}'
{
"name": "device",
"version": "1.0.0",
"wireless": [
{
"name": "wlan1_wifi",
"type": "5Ghz",
"ssid": "wifi"
},
{
"name": "wlan2_link",
"type": "2Ghz",
"ssid": "link"
}
]
}
Now there's a problem. I need to assign an address to the "wireless" array. The address is stored in "address" array.
So the question: is there a way of finding the right json object in "address" based on "name" (in wireless array) and "interface" (in address array) for every json object in "wireless" array and then assigning "address" to it?
The final result should look like this:
{
"name": "device",
"version": "1.0.0",
"wireless": [
{
"name": "wlan1_wifi",
"type": "5Ghz",
"ssid": "wifi",
"address": "10.1.2.3"
},
{
"name": "wlan2_link",
"type": "2Ghz",
"ssid": "link",
"address": "10.1.2.5"
}
]
}
Answer:
Here's my answer based on the answer from #peak. Instead of copying the content of .wireless and then using map, I'm cherry picking the keys that I want to include only. This also allows me to position "address" how ever I want.
(INDEX(.address[]; .interface)) as $dict
| {name: .name, version: .version,
wireless: [.wireless[] | {name, address: ($dict[.name]|.address), type, ssid}]}
The following produces the output as originally requested:
(.wireless[].name) as $name
| .address[]
| select(.interface == $name)
| { wireless: {name: $name, address}}
However the above filter could potentially produce more than one result, so you might want to make modifications accordingly.
Revised revised requirements
If your jq has INDEX/2 (which was only made available AFTER jq 1.5 was released), you can simply use it to create a lookup table:
(INDEX(.address[]; .interface)) as $dict
| {name,
version,
wireless: (.wireless
| map(. + {address: ($dict[.name]|.address) }) ) }
Or (depending perhaps on the exact requirements):
(INDEX(.address[]; .interface)) as $dict
| del(.address)
| .wireless |= map(. + {address: ($dict[.name]|.address) })
If your jq does not have INDEX/2, then you could easily adapt the above (using reduce), or even more easily snarf the def of INDEX/2 from https://github.com/stedolan/jq/blob/master/src/builtin.jq

ARM listKeys() Function - How to retrieve OMS/OpsInsight Workspace keys?

As part of a template I want to retrieve the SharedKeys of an OMS / Operational Insights Workspace, rather than having to pass it in as a parameter.
Is this possible? I'm following the documentation here
It does not appear that the Microsoft.OperationalInsights/workspaces/ resource provider has any list* provider operations, and I can't find any reference for other:
Get-AzureRmProviderOperation -OperationSearchString * | where {$_.Operation -like "*operational*sharedkeys*"} | FT Operation
Microsoft.OperationalInsights/workspaces/sharedKeys/action
My desired usage:
"variables": { workspaceKey: "[listKeys(parameters('workspaceResourceId'), '2015-05-01-preview').primarySharedKey]" }
In the meantime, assuming this isn't actually supported, I added a request for it on the Log Analytics UserVoice site
Per Ryan Jones, [listKeys()] against the OMS Workspace will work as expected and return a JSON object with primarySharedKey & secondarySharedKey properties:
"outputs": {
"listKeys": {
"value": "[listKeys(parameters('workspaceResourceId'), '2015-11-01-preview')]",
"type": "object"
}
}
yields:
{
"primarySharedKey":"",
"secondarySharedKey":""
}
Important Caveat:
listKeys() can not be specified in the variables section of an ARM template, since it derives its value from a runtime state.
See this blog post for how to use a Linked Template, specified as a resource, in order to retrieve the output value and assign it to a property in another resource.
Alternatively, you can use it directly. Here is my final template:
(don't actually keep the keys in the output!)
{
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceResourceId": { "type": "string" },
"virtualMachines": { "type": "array" }
},
"variables": {
"extensionType": {
"Windows": "MicrosoftMonitoringAgent",
"Linux": "OmsAgentForLinux"
}
},
"resources": [
{
"copy": {
"name": "VMMonitoringExtensionsCopy",
"count": "[length(parameters('virtualMachines'))]"
},
"type": "Microsoft.Compute/virtualMachines/extensions",
"apiVersion": "2015-05-01-preview",
"location": "[parameters('virtualMachines')[copyIndex()].location]",
"name": "[concat(parameters('virtualMachines')[copyIndex()].name, '/Microsoft.EnterpriseCloud.Monitoring')]",
"properties": {
"publisher": "Microsoft.EnterpriseCloud.Monitoring",
"type": "[variables('extensionType')[parameters('virtualMachines')[copyIndex()].osType]]",
"typeHandlerVersion": "1.0",
"autoUpgradeMinorVersion": true,
"settings": {
"workspaceId": "[reference(parameters('workspaceResourceId'), '2015-11-01-preview').customerId]"
},
"protectedSettings": {
"workspaceKey": "[listKeys(parameters('workspaceResourceId'), '2015-11-01-preview').primarySharedKey]"
}
}
}
],
"outputs": {
"workspaceCustomerId": {
"value": "[reference(parameters('workspaceResourceId'), '2015-11-01-preview').customerId]",
"type": "string"
},
"workspacePrimarySharedKey": {
"value": "[listKeys(parameters('workspaceResourceId'), '2015-11-01-preview').primarySharedKey]",
"type": "securestring"
},
"workspaceSecondarySharedKey": {
"value": "[listKeys(parameters('workspaceResourceId'), '2015-11-01-preview').secondarySharedKey]",
"type": "securestring"
}
}
}
The array parameter virtualMachines follows this schema:
[
{ "name": "", "location": "", "osType": "" }
]
listKeys requires that you put the resource type in. So have you tried this?
"variables": { workspaceKey: "[listKeys(resourceId('Microsoft.OperationalInsights/workspaces', parameters('workspaceResourceId'), '2015-05-01-preview').primarySharedKey]" }
Unfortunately, atm there is nothing at all in the Azure quickstart repo on that resource so I'm not 100% sure...
But passing it in as a parameter would be fine. You could do this... In your deployment script, before you run New-AzureRmResourceGroupDeployment, create/use existing workspace, get key, pass in as param, create primarySharedKey as a param in the template:
$workSpace = Get-AzureRmOperationalInsightsWorkspace -ResourceGroupName $RGName -Name $workSpaceName -ErrorAction SilentlyContinue
if($workSpace -eq $null){
New-AzureRmOperationalInsightsWorkspace -ResourceGroupName $RGName -Name $workSpaceName -Location $Location
}
$keys = Get-AzureRmOperationalInsightsWorkspaceSharedKeys -ResourceGroupName $RGName -Name $workSpaceName
New-AzureRmResourceGroupDeployment <other stuff here> -primarySharedKey $keys.PrimarySharedKey

How to convert complex JSON to CSV using JQ 1.4

I am using JQ 1.4 on Windows 64 bit machine.
Below are the contents of input file IP.txt
{
"results": [
{
"name": "Google",
"employees": [
{
"name": "Michael",
"division": "Engineering"
},
{
"name": "Laura",
"division": "HR"
},
{
"name": "Elise",
"division": "Marketing"
}
]
},
{
"name": "Microsoft",
"employees": [
{
"name": "Brett",
"division": "Engineering"
},
{
"name": "David",
"division": "HR"
}
]
}
]
}
{
"results": [
{
"name": "Amazon",
"employees": [
{
"name": "Watson",
"division": "Marketing"
}
]
}
]
}
File contains two "results". 1st result containts information for 2 companies: Google and Microsoft. 2nd result contains information for Amazon.
I want to convert this JSON into csv file with company name and employee name.
"Google","Michael"
"Google","Laura"
"Google","Elise"
"Microsoft","Brett"
"Microsoft","David"
"Amazon","Watson"
I am able to write below script:
jq -r "[.results[0].name,.results[0].employees[0].name]|#csv" IP.txt
"Google","Michael"
"Amazon","Watson"
Can someone guide me to write the script without hardcoding the index values?
Script should be able generate output for any number results and each cotaining information of any number of companies.
I tried using below script which didn't generate expected output:
jq -r "[.results[].name,.results[].employees[].name]|#csv" IP.txt
"Google","Microsoft","Michael","Laura","Elise","Brett","David"
"Amazon","Watson"
You need to flatten down the results first to rows of company and employee names. Then with that, you can convert to csv rows.
map(.results | map({ cn: .name, en: .employees[].name } | [ .cn, .en ])) | add[] | #csv
Since you have a stream of inputs, you'll have to slurp (-s) it in. Since you want to output csv, you'll want to use raw output (-r).