JSON file specific keywork via powershell - json

I want to get lines which are serviceArea equal Skype like below.
{
"id": 127,
"serviceArea": "Skype",
"serviceAreaDisplayName": "Skype for Business Online and Microsoft Teams",
"urls": [
"*.skype.com"
],
"tcpPorts": "80,443",
"expressRoute": false,
"category": "Default",
"required": true
},
{
"id": 128,
"serviceArea": "Common",
"serviceAreaDisplayName": "Microsoft 365 Common and Office Online",
"urls": [
"*.config.office.net",
"*.manage.microsoft.com"
],
"tcpPorts": "443",
"expressRoute": false,
"category": "Default",
"required": true
},
{
"id": 130,
"serviceArea": "Common",
"serviceAreaDisplayName": "Microsoft 365 Common and Office Online",
"urls": [
"lpcres.delve.office.com"
],
"tcpPorts": "443",
"expressRoute": false,
"category": "Default",
"required": true
},
{
"id": 146,
"serviceArea": "Skype",
"serviceAreaDisplayName": "Skype for Business Online and Microsoft Teams",
"urls": [
"statics.teams.microsoft.com"
],
"tcpPorts": "443",
"expressRoute": false,
"category": "Default",
"required": true
}
My desired output :
*.skype.com
statics.teams.microsoft.com

Assuming your JSON is wrapped in an array (as mentioned in comments above), you could do something like this to filter the data.
param(
$jsonFile = "$PSScriptRoot\data.json",
$likeFilter = "*Skype*"
)
$convertedObjects = Get-Content -Path $jsonFile | ConvertFrom-Json
$result = $convertedObjects | Where-Object { $_.serviceArea -like $likeFilter }
$result

Your JSON is invalid due to is has no root element. As I see, it is array (list)
$text = #'
[
<your text here>
]
'#
After that, you can select urls
$result = $text |
ConvertFrom-Json |
Where-Object { $_.serviceArea -eq 'Skype' } |
ForEach-Object { return $_.urls }

Related

Converting JSON to NDJSON with Powershell

I'm trying to convert a text file that has an array of json objects to an NDJSON formatted file for another team to consume.
I've almost got it, except for one problem. I have an array of objects nested inside the JSON (which then has nested arrays and objects inside of it, the structure gets pretty complex, I'll include a sample below) and for whatever reason, when I use ConvertFrom-JSON it drops this nested array and in my output, I end up with a blank string for that key, instead of the nested array object. I tried using the -Depth flag but when I do that my output file ends up blank, which doesn't make a ton of sense to me. I don't have a whole lot of experience with powershell, so I'm not really sure where I'm going wrong here.
Code:
$JSONSourceFile = Get-Content -Path "input/sample.json" | ConvertFrom-JSON
$NDJSONTargetFile = "output/sample.json"
New-Item $NDJSONTargetFile -ItemType file
for ( $i = 0 ; $i -lt $JSONSourceFile.Length ; $i++) {
$item = $JSONSourceFile.item($i)
$row = ($item | ConvertTo-JSON -Compress)
Add-Content $NDJSONTargetFile $row
}
Input File:
[
{
"id": "1",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak2354",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "Y"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "N"
}
]
}
]
}
]
},
{
"id": "2",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak1234",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "Y"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "Y"
}
]
}
]
}
]
},
{
"id": "3",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak5678",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "N"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "N"
}
]
}
]
}
]
}
]
And then when I convert it to the output, this is what I get:
{"id":"1","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak2354","Preferences":""}]}
{"id":"2","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak1234","Preferences":""}]}
{"id":"3","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak5678","Preferences":""}]}
Thanks to the comment from Doug Maurer I figured it out, I was adding -Depth to my ConvertFrom-Json command when it should have been in my ConvertTo-Json command. This is the final script and what it gives me:
$JSONSourceFile = Get-Content -Path "input/sample.json" | ConvertFrom-JSON
$NDJSONTargetFile = "output/sample.json"
New-Item $NDJSONTargetFile -ItemType file
for ( $i = 0 ; $i -lt $JSONSourceFile.Length ; $i++) {
$item = $JSONSourceFile.item($i)
$row = ($item | ConvertTo-JSON -Compress -Depth 20)
Add-Content $NDJSONTargetFile $row
}
and the output:
{"id":"1","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak2354","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"Y"},{"ChannelID":"2","ChannelName":"Text","Preference":"N"}]}]}]}
{"id":"2","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak1234","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"Y"},{"ChannelID":"2","ChannelName":"Text","Preference":"Y"}]}]}]}
{"id":"3","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak5678","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"N"},{"ChannelID":"2","ChannelName":"Text","Preference":"N"}]}]}]}

Create a json document based on existing document with powershell 5.1

I'm trying to build a json document which takes it's structure, column names and some values from an already existing base json document, and sets other values to variables assigned via powershell 5.1 in a foreach-object loop.
I tried editing a copy of the base json and creating a new one entirely and I can get close to what I want but not exactly.
A cut down example json document is:
[
{
"name": "Application1",
"tags": ["monitor"],
"description": "Get information about application 1.",
"query":
[
{
"target": {
"version": ["v1","v2","v3","v4","v5"],
"platform": ["Windows","Linux"]
},
"implementation": {
"action": "do something"
}
}
]
},
{
"name": "Application2",
"tags": ["monitor"],
"description": "Get information about application 2",
"query":
[
{
"target": {
"version": ["v1","v2","v3"],
"platform": ["Windows"]
},
"implementation": {
"action": "do something for v1, v2 and v3"
}
},
{
"target": {
"version": ["v4","v5"],
"platform": ["Windows"]
},
"implementation": {
"action": "do something for v4 and v5"
}
}
]
}
]
My script currently looks like:
$BaseJson = (Get-Content "$SourcePath\probes.json" | ConvertFrom-Json)
$ClientConfig = #"
{
"targetHost": "$DigitalSymphonyHost",
"targetPort": "$DigitalSymphonyPort",
"source": "$TargetSqlInstance",
"sensor": {}
}
"#
$ClientJson = ConvertFrom-Json -InputObject $ClientConfig
$BaseJson | Where-Object {$_.tags -contains "monitor"} | ForEach-Object {
# For each sensor in the base-sensors json document
$SensorName = $_.name
$Severity = 2
$Version = $_.query.target.version
$Platform = $_.query.target.platform
$Query = $_.query.implementation.action
$Sensor =#"
{
"severity": "$Severity",
"sensorId": "$SensorId",
"type": "$Type",
"target": {
"version": "$Version",
"platform": "$Platform",
"engineEdition": "$Edition"
},
"implementation": {
"query": "$Query"
}
}
"#
$ClientJson.sensor | Add-Member -Name $SensorName -Value (ConvertFrom-Json $Sensor) -MemberType NoteProperty
}
$ClientJson | ConvertTo-Json | Out-File "$DestinationPath\client-sensors.json"
My desired result is to add the $SensorId as per:
[
{
"targetHost": "servername",
"targetPort": "port",
"source": "servername",
"sensor": [{
"name": "Application1",
"sensorId": "<value_from_$SensorId>",
"tags": [],
"description": "Get information about application 1.",
"query":
[
{
"target": {
"version": ["v1","v2","v3","v4","v5"],
"platform": ["Windows","Linux"]
},
"implementation": {
"action": "do something"
}
}
]
},
{
"name": "Application2",
"sensorId": "<value_from_$SensorId>",
"tags": [],
"description": "Get information about application 2",
"query":
[
{
"target": {
"version": ["v1","v2","v3"],
"platform": ["Windows"]
},
"implementation": {
"action": "do something for v1, v2 and v3"
}
},
{
"target": {
"version": ["v4","v5"],
"platform": ["Windows"]
},
"implementation": {
"action": "do something for v4 and v5"
}
}
]
}]
}
]
My current result is
{
"targetHost": "servername",
"targetPort": "port",
"source": "server",
"sensor": {
"applicationname1": {
"tags": ["monitor],
"description": "Get information about application 2",
"sensorId": "420",
"target": "#{version=v1,v2,v3,v4,v5; platform=Windows Windows;}",
"action": "do something for v4 and v5"
}
}
}
I'm not sure where you're getting sensorID, but I'd do this more like the below:
param(
$SourcePath = $PSScriptRoot,
$DestinationPath = $PSScriptRoot,
$DigitalSymphonyHost = 'SomeHost',
$DigitalSymphonyPort = '8765',
$TargetSqlInstance
)
$BaseObj = (Get-Content "$SourcePath\probes.json" | ConvertFrom-Json)
$Client = [PSCustomObject] #{
targetHost = $DigitalSymphonyHost
targetPort = $DigitalSymphonyPort
source = $TargetSqlInstance
sensor = #()
}
$BaseObj | Where-Object {$_.tags -contains "monitor"} | ForEach-Object {
$sensor = $_
# TODO: How are you getting sensorId?
$sensor | Add-Member -Name 'sensorId' -Value 777 -MemberType NoteProperty
$Client.sensor += $sensor
}
ConvertTo-Json #($Client) -Depth 10 | Out-File "$DestinationPath\client-sensors.json" -Force
That will get you the JSON you're looking for.

Get the node value from json in powershell

I have json body in powershell variable and want to get value from it.
Json Body :
"value": [
{
"id": 1,
"scope": "3e93f9e6-f427-48e1-b37b-994d196e1121",
"name": "Default",
"isHosted": false,
"poolType": "automation",
"size": 3
},
{
"id": 2,
"scope": "3e93f9e6-f427-48e1-b37b-994d196e1121",
"name": "Hosted",
"isHosted": true,
"poolType": "automation",
"size": 10
},
{
"id": 4,
"scope": "3e93f9e6-f427-48e1-b37b-994d196e1121",
"name": "Hosted VS2017",
"isHosted": true,
"poolType": "automation",
"size": 10
}]
I want to get the id value where name=Hosted VS2017 i.e 4
powershell :
Write-Host "json body:" $projects
$id = $projects.value | where { $_.name -eq "Hosted VS2017" }
Write-Host "id :" $id
You need to convert your JSON into a powershell object before you can work with it:
$projects = $projects | ConvertFrom-Json
Then you can access its members:
#requires -Version 3
$projects.value | ? name -eq 'Hosted VS2017'

ARM Template DSC: configuration does not 'see' protectedSettings.configurationArguments

I need to protect a DSC configuration parameter ([pscredential]RegistrationKey), so I have put it under "settings.protectedSettings.configurationData" thus:
"protectedSettings": {
"configurationArguments": {
"RegistrationKey": {
"UserName": "PLACEHOLDER_DONOTUSE",
"Password": "[parameters('dscAutomationRegistrationKey')]"
}
},
"configurationUrlSasToken": "[parameters('artifactsLocationSasToken')]"
}
I get the error:
"VM has reported a failure when processing extension 'Microsoft.Powershell.DSC'. Error message: \"The DSC Extension failed to execute: Mandatory
parameter RegistrationKey is missing.
If I move RegistrationKey out of "settings.protectedSettings.configurationArguments", into "settings.configurationArguments", it works, therefore, I assume there is nothing wrong with the syntax, so I believe it is to do with PsDscAllowPlainTextPassword = $true that wasn't included in the DSC configuration.
(I tried to include the configuration block in the PS1 file, but this threw an error, suggesting this can't be done)
I have now written a configurationdata .psd1 file, containing the following:
$ConfigData = #{
AllNodes = #(
#{
NodeName = "*"
PsDscAllowPlainTextPassword = $true
}
)
}
and referenced it in settings.configurationdata.url.
This now results in the same error as before: VM has reported a failure...
The ARM template is called from PowerShell:
$oAutomationAccount = Get-AzureRmAutomationAccount -ResourceGroupName $AAresourceGroupName -Name $AutomationAccountName
$RegistrationInfo = $oAutomationAccount | Get-AzureRmAutomationRegistrationInfo
$DscRegKeyString = $RegistrationInfo.PrimaryKey
$ssDscAutomationRegistrationKey = (ConvertTo-SecureString -string $DscRegKeyString -AsPlainText -Force)
#Automation Account EndPoint Uri
$DscRegistrationUrl = $RegistrationInfo.Endpoint
$params = #{
artifactsLocationSasToken = $TemplateSas
vmName = "XYZ"
dscAutomationRegistrationKey = $ssDscAutomationRegistrationKey
dscAutomationRegistrationUrl = $DscRegistrationUrl
dscNodeConfigurationName = "CreateAFolder.localhost"
dscTimeStamp = (Get-Date -f "MM/dd/yyyy H:mm:ss tt") #"MM/dd/yyyy H:mm:ss tt"
dscResourceUrl = $DscResourceUrl
dscConfigurationUrl = $DscConfigurationUrl
dscResourceScript = $DscResourceScriptName
dscResourceFunction = "ConfigureLCMforAAPull"
#sequenceId = $sequenceId
}
New-AzureRmResourceGroupDeployment #params `
-Name "$TemplateInstance-$branch" `
-ResourceGroupName $DeploymentResourceGroup.ResourceGroupName `
-Mode Incremental `
-DeploymentDebugLogLevel All `
-TemplateUri $TemplateUri `
-Verbose
Where I believe the parameters are being passed as the correct types.
What am I doing wrong?
Reference template: https://github.com/Azure/azure-quickstart-templates/blob/master/dsc-extension-azure-automation-pullserver/azuredeploy.json
Updated to use a newer DSC schema:https://blogs.msdn.microsoft.com/powershell/2016/02/26/arm-dsc-extension-settings/
this is the template I've been using for node onboarding:
{
"name": "xxx",
"type": "Microsoft.Compute/virtualMachines/extensions",
"location": "[parameters('location')]",
"apiVersion": "2015-06-15",
"dependsOn": [
"xxx"
],
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "2.22",
"autoUpgradeMinorVersion": false,
"protectedSettings": {
"Items": {
"registrationKeyPrivate": "[parameters('registrationData')]"
}
},
"settings": {
"ModulesUrl": "https://github.com/Azure/azure-quickstart-templates/raw/master/dsc-extension-azure-automation-pullserver/UpdateLCMforAAPull.zip",
"SasToken": "",
"ConfigurationFunction": "UpdateLCMforAAPull.ps1\\ConfigureLCMforAAPull",
"Properties": [
{
"Name": "RegistrationKey",
"Value": {
"UserName": "PLACEHOLDER_DONOTUSE",
"Password": "PrivateSettingsRef:registrationKeyPrivate"
},
"TypeName": "System.Management.Automation.PSCredential"
},
{
"Name": "RegistrationUrl",
"Value": "xxx",
"TypeName": "System.String"
},
{
"Name": "NodeConfigurationName",
"Value": "xxx",
"TypeName": "System.String"
},
{
"Name": "ConfigurationMode",
"Value": "ApplyAndMonitor",
"TypeName": "System.String"
},
{
"Name": "ConfigurationModeFrequencyMins",
"Value": 15,
"TypeName": "System.Int32"
},
{
"Name": "RefreshFrequencyMins",
"Value": 30,
"TypeName": "System.Int32"
},
{
"Name": "RebootNodeIfNeeded",
"Value": true,
"TypeName": "System.Boolean"
},
{
"Name": "ActionAfterReboot",
"Value": "ContinueConfiguration",
"TypeName": "System.String"
},
{
"Name": "AllowModuleOverwrite",
"Value": true,
"TypeName": "System.Boolean"
},
{
"Name": "Timestamp",
"Value": "MM/dd/yyyy H:mm:ss tt",
"TypeName": "System.String"
}
]
}
}
}
I know its using an old format, but that works so, meh.

Azure functions & Powershell: get response in plain JSON

Trying out some new features in Azure; Azure Functions running Powershell. One thing I'm struggling with for a while: I cannot get the output in plain JSON, instead in seems formatted and includes escape characters and is messed up double quoting.
function.json:
{
"bindings": [
{
"name": "req",
"type": "httpTrigger",
"direction": "in",
"webHookType": "genericJson",
"methods": [
"post",
"head",
"trace"
]
},
{
"name": "res",
"type": "http",
"direction": "out"
}
],
"disabled": false
}
run.ps1:
$ApiURI="http://avwx.rest/api/metar.php?station=ymml&format=JSON"
$Result = Invoke-RestMethod -Uri $ApiURI
Out-File -Encoding Ascii -FilePath $res -inputObject $Result
Results in following response:
"\r\n\r\nAltimeter : 1015\r\nCloud-List : {FEW 035}\r\nDewpoint : 11\r\nFlight-Rules : VFR\r\nOther-List : {}\r\nRaw-Report : YMML 210900Z 17010KT 9999 FEW035 19/11 Q1015 NOSIG\r\nRemarks : NOSIG\r\nRemarks-Info : \r\nRunway-Vis-List : {}\r\nStation : YMML\r\nTemperature : 19\r\nTime : 210900Z\r\nUnits : #{Altimeter=hPa; Altitude=ft; Temperature=C; Visibility=m; \r\n Wind-Speed=kt}\r\nVisibility : 9999\r\nWind-Direction : 170\r\nWind-Gust : \r\nWind-Speed : 10\r\nWind-Variable-Dir : {}\r\n\r\n\r\n\r\n"
But what I expect is something like:
{
"Altimeter": "1016",
"Cloud-List": [],
"Dewpoint": "12",
"Flight-Rules": "VFR",
"Other-List": [],
"Raw-Report": "YMML 210930Z 17011KT CAVOK 18/12 Q1016 NOSIG",
"Remarks": "NOSIG",
"Remarks-Info": {},
"Runway-Vis-List": [],
"Station": "YMML",
"Temperature": "18",
"Time": "210930Z",
"Units": {
"Altimeter": "hPa",
"Altitude": "ft",
"Temperature": "C",
"Visibility": "m",
"Wind-Speed": "kt"
},
"Visibility": "9999",
"Wind-Direction": "170",
"Wind-Gust": "",
"Wind-Speed": "11",
"Wind-Variable-Dir": []
}
What could be wrong here?
I'm not sure how you got that result, since Invoke-RestMethod store the data in the ResponseData property.
When you use Invoke-RestMethod it will convert the json data to a Powershell Object. In your run.ps1 it looks like you are going to save the PS Object to a file, not the JSON data.
If you are not going to modify the JSON data, it's really no use to convert it to a PSObject. So you could use Invoke-webrequest instead.
But in case you want to modify it.
$uri = "http://avwx.rest/api/metar.php?station=ymml&format=JSON"
$request = Invoke-RestMethod -Uri $uri
#Get the result and convert it back to JSON
$Result = $request.responseData | ConvertTo-Json
$Result | Out-File .\data.json -Encoding Ascii
#David - Yes, the following works for me in Azure Functions.
a. run.ps1 file content
$ApiURI="http://avwx.rest/api/metar.php?station=ymml&format=JSON"
$Result = Invoke-RestMethod -Uri $ApiURI | ConvertTo-Json
Out-File -Encoding Ascii -FilePath $res -inputObject $Result
b. Request result both in Azure Functions Portal and Postman.
{
"Altimeter": "1016",
"Cloud-List": [
[
"FEW",
"016"
]
],
"Dewpoint": "13",
"Flight-Rules": "VFR",
"Other-List": [],
"Raw-Report": "YMML 212100Z 28005KT 9999 FEW016 15/13 Q1016 NOSIG",
"Remarks": "NOSIG",
"Remarks-Info": {},
"Runway-Vis-List": [],
"Station": "YMML",
"Temperature": "15",
"Time": "212100Z",
"Units": {
"Altimeter": "hPa",
"Altitude": "ft",
"Temperature": "C",
"Visibility": "m",
"Wind-Speed": "kt"
},
"Visibility": "9999",
"Wind-Direction": "280",
"Wind-Gust": "",
"Wind-Speed": "05",
"Wind-Variable-Dir": []
}