I've seen a lot of questions about JSON and PowerShell these past hours and none helped me find a solution to this particular problem. And I'm sure it's something easy.
I want to extract all the url fields of the plugins objects in this JSON object (original URL is this: https://updates.jenkins.io/update-center.json):
{
"connectionCheckUrl": "http://www.google.com/",
"core": {
...
},
"deprecations": {
...
},
"generationTimestamp": "2021-05-19T12:16:52Z",
"id": "default",
"plugins": {
"42crunch-security-audit": {
"buildDate": "Oct 06, 2020",
"defaultBranch": "master",
"dependencies": [
...
],
"developers": [
...
],
"excerpt": "Performs API contract security audit to get a detailed analysis of the possible vulnerabilities and other issues in the API contract.",
"gav": "io.jenkins.plugins:42crunch-security-audit:3.8",
"issueTrackers": [
...
],
"labels": [
...
],
...
"title": "42Crunch REST API Static Security Testing",
"url": "http://archives.jenkins-ci.org/plugins/42crunch-security-audit/3.8/42crunch-security-audit.hpi",
},
"AnchorChain": {
...
"url": "http://archives.jenkins-ci.org/plugins/AnchorChain/1.0/AnchorChain.hpi",
...
},
... many hundreds more ...
}
...
}
The plugins object contains one object per plugin, where the plugin's name is the object's key. So I somehow have to iterate over all plugin objects and look for the url property.
I want/have to do this using PowerShell (v5.1) but cannot find an easy way. Here is where I am stuck:
$all = (Get-Content(".\update-center.json") | convertfrom-json)
$all.gettype().fullname
$plugins = $all.plugins
$plugins.gettype().fullname
I get this result:
System.Management.Automation.PSCustomObject
System.Management.Automation.PSCustomObject
And now I hope to iterate over the individual plugin objects and simply get the url key's property, but I'm stuck:
$plugins | get-member -MemberType NoteProperty | foreach name | foreach $plugins.$_.url
The get-member is supposed to get the individual plugins I suppose, but hours of poring over PowerShell documentation have clearly fried my brain. Help! :-)
Import your JSON as you did for $all
Then, use $all.plugins | gm -MemberType Properties | select -expandproperty Name | %{ $all.plugins.$_.url} to get your list of urls
I think this is what you're looking for, not exactly sure. Correct me if I'm wrong.
$Json = Invoke-RestMethod https://updates.jenkins.io/update-center.json
$Json = $Json -replace '^updateCenter.post\(|\);$' | ConvertFrom-Json
$plugins = $Json.plugins
foreach($prop in $plugins.psobject.properties.name)
{
$plugins.$prop.url
}
Output
https://updates.jenkins.io/download/plugins/testingbot/1.16/testingbot.hpi
https://updates.jenkins.io/download/plugins/testinium/1.0/testinium.hpi
https://updates.jenkins.io/download/plugins/testlink/3.16/testlink.hpi
https://updates.jenkins.io/download/plugins/testng-plugin/1.15/testng-plugin.hpi
https://updates.jenkins.io/download/plugins/testodyssey-execution/2.1.5/testodyssey-execution.hpi
https://updates.jenkins.io/download/plugins/testopia/1.3/testopia.hpi
https://updates.jenkins.io/download/plugins/testproject/2.10/testproject.hpi
https://updates.jenkins.io/download/plugins/testquality-updater/1.3/testquality-updater.hpi
https://updates.jenkins.io/download/plugins/testsigma/1.3/testsigma.hpi
....
....
....
Related
I have two JSON files and want to transfer collection of objects from one file to another. Suppose, the from.json file contains property which represents collection of clients:
"Clients":
[
{
"Name": "Name1",
"Age": "12"
},
{
"Name": "Name2",
"Age": "14"
}
]
to.json file contains an empty collection, "Objects: []" ,which must be filled with objects from from.json. Each objects in toJson variable must contain additional property - Id, so eventually, my "to.json" file should look like this:
"Objects":
[
{
"Id": "{new-id}",
"Name": "Name1",
"Age": "12"
},
{
"Id": "{new-id}",
"Name": "Name1",
"Age": "12"
}
]
I've converted two files into variables:
$fromJson = (Get-Content -Raw -Path {fromPath}) | ConvertFrom-Json
$toJson = (Get-Content -Raw -Path {toPath}) | ConvertFrom-Json
I know that objects from fromJson to toJson can be transferred in the following manner:
toJson.Objects += fromJson.Clients, but that's not enough in my case. I think that it could be done by iterating through fromJson.Clients array but have no idea how to create an object and add it into toJson.Objects collection.
Here's a more efficient solution, based on:
Use of a calculated property with Select-Object, which allows you to place the new property first in the output objects.
Instead of building the array one by one with += (which is inefficient, because a new array must technically be created behind the scenes in every iteration), the solution below lets PowerShell collect the output objects of the Select-Object call in an array automatically (the [array] type constraint is needed to ensure that an array is created even if only one object happens to be output.)
# Sample input.
$fromJson = ConvertFrom-Json '{"Clients":[{"Name":"Name1","Age":"12"},{"Name":"Name2","Age":"14"}]}'
$toJson = ConvertFrom-Json '{ "Objects": [] }'
[array] $toJson.Objects =
$fromJson.Clients |
Select-Object #{ Name='Id'; Expression = { [string] (New-Guid) } }, *
$toJson | ConvertTo-Json -Depth 3 # append | Set-Content as needed.
Kind of new to the PowerShell, but after a bit of investigation came up with the following solution:
fromJson.Clients | ForEach-Object {
$_ | Add-Member -MemberType NoteProperty -Name 'Id' -Value ([guid]::NewGuid().Guid.ToString())
$toJson += $_
}
...
$toJson | ConvertTo-Json | Out-File {to.json_path}
Frankly, don't know if that is a 'proper' way to do that, but generally it works for that particular case. For now, see no other solution.
I am reading data from a json file via powershell, with the ultimate goal of updating said file. I need to split the data in a chunk I want to keep and a chunk I want to update, and to complicate the matter further, the place where I need to split the text varies throughout a foreach-loop, thus I need that part to come from a variable.
I have tried .split/-split and .replace/-replace in numerous configurations, but it seems this is harder than one would assume in powershell.
All the below files in the same folder.
Json (json.json):
{
"Section1": {
"Heading1": [
"Thing1",
"Thing2"
]
},
"Section2": [
"Thing1",
"Thing2"
]
}
Powershell (powershell.ps1):
$originalJsonString = Get-Content -Path ".\json.json"
$SplitTarget = "Section2"
$JsonString = {This is the part that I am iffy about}
Write-Output $JsonString
The output I want from the above would be
{
"Section1": {
"Heading1": [
"Thing1",
"Thing2"
]
},
I have tried just about everything I can think of in relation to split and replace, but yet the solution alludes me.
Note in the solution above, that it is an important factor that $originalJsonString is split (or whatever) by $SplitTarget, and not "Section2", as that is also a factor in my equation.
This is my first time asking, so if I am doing something wrong I apologise.
Thank you.
Edit:
It is only fair that I add the reason I don't convert to an object.
The syntax which powershell exports when converting json to an object and back is undesirable for my use.
However, if using an object is the ONLY way, and slitting is out of the question, then another solution must be found.
Thank you.
Edit:
If objects was the way to go, I ended up finding a way more complicated way to format the .json the way I wanted it.
Thanks everyone.
What you should do is read the json file in using :
$json = Get-Content .\json.json -raw | ConvertFrom-Json
I faked that here using a 'here-string':
$json = #"
{
"Section1": {
"Heading1": [
"Thing1",
"Thing2"
]
},
"Section2": [
"Thing1",
"Thing2"
]
}
"# | ConvertFrom-Json
Next, define what you want to keep or update:
$sectionToKeep = "Section1"
$sectionToUpdate = "Section2"
To see what is in there use $json.$sectionToKeep | ConvertTo-Json
{
"Heading1": [
"Thing1",
"Thing2"
]
}
Next, update $section2 leaving everything else untouched. I am writing an object that stores an array, just like in $sectionToKeep:
$json.$sectionToUpdate = #{'Heading2' = 'Thing3', 'Thing4'}
and finally output (or write back to file) the new complete json:
$json | ConvertTo-Json
Using your example gives you this:
{
"Section1": {
"Heading1": [
"Thing1",
"Thing2"
]
},
"Section2": {
"Heading2": [
"Thing3",
"Thing4"
]
}
}
Hope that helps
Not sure to really understand your question, but I would convert the Json to an object and then filter the sub data and create a file again
$obj = Get-Content json.json -raw | ConvertFrom-Json
If you are looking to just isolate a section and create a new json file, then you can use the following:
$file = Get-Content .\json.json | ConvertFrom-Json
$file | Select-Object Section1 | ConvertTo-Json | Set-Content New.json
I need to get only some Information out of my JSON content, but with normal select-object and where-object my PowerShell prompt gives my nothing.
What I do:
I get a JSON output from a webpage and then only need the .Content.
$get_all_attributes = $result.Content | Out-String | ConvertFrom-Json | Select Attributes
When asking PowerShell to give me one Particular Object like $get_all_attributes.Attributes.Slot1 everything is fine.
But now I need to get all Slots (Slot1 - SlotX) without the Bif (eg Slot1 but not Slot1Bif).
Afterwards I like to find all disabled ones.
But for now I even do net get the Slots.
I converted it in some ways from and to Json with String and whatever but now I'm kinda stuck.
Nice looking JSON
{
"Attributes": {
"AcPwrRcvry": "Last",
"AcPwrRcvryDelay": "Immediate",
"AesNi": "Enabled",
"AssetTag": "",
"BootMode": "Uefi",
"BootSeqRetry": "Enabled",
"CollaborativeCpuPerfCtrl": "Disabled",
"ConTermType": "Vt100Vt220",
"ControlledTurbo": "Disabled",
"Slot1": "Enabled",
"Slot1Bif": "DefaultBifurcation",
"Slot2": "Enabled",
"Slot2Bif": "DefaultBifurcation",
"Slot3": "Enabled",
"Slot3Bif": "DefaultBifurcation",
"Slot4": "Enabled",
"Slot4Bif": "DefaultBifurcation",
"Slot5": "Enabled",
"Slot5Bif": "DefaultBifurcation",
"Slot6": "Enabled",
"Slot6Bif": "DefaultBifurcation",
"Slot7": "Enabled",
"Slot7Bif": "DefaultBifurcation"
}
}
My Converted Stuff
$get_all_attributes | FL
Attributes : #{AcPwrRcvry=Last; AcPwrRcvryDelay=Immediate; AesNi=Enabled; AssetTag=; BootMode=Uefi; BootSeqRetry=Enabled; CollaborativeCpuPerfCtrl=Disabled;
ConTermType=Vt100Vt220; ControlledTurbo=Disabled; CorrEccSmi=Enabled; CpuInterconnectBusLinkPower=Enabled; CurrentEmbVideoState=Enabled;
DcuIpPrefetcher=Enabled;Slot1=Enabled; Slot1Bif=DefaultBifurcation; Slot2=Enabled; Slot2Bif=DefaultBifurcation; Slot3=Enabled; Slot3Bif=DefaultBifurcation; Slot4=Enabled;
Slot4Bif=DefaultBifurcation; Slot5=Enabled; Slot5Bif=DefaultBifurcation; Slot6=Enabled; Slot6Bif=DefaultBifurcation; Slot7=Enabled;
Slot7Bif=DefaultBifurcation}
You're almost there, just use the switch "ExpandProperty".
$get_all_attributes = $result.Content | Out-String | ConvertFrom-Json | Select -ExpandProperty Attributes
After, this, the easiest way is to simply select the property you're interested in to get all the fields...
$get_all_attributes.Attributes.BootSeqRetry
... or get more granular for a specific sub-property:
$get_all_attributes.Attributes.BootSeqRetry
(In this case, it returns Enabled)
The following code should solve your problem.
$attributes = $get_all_attributes.Attributes;
$filteredAttributes = $attributes | Select-Object -Property "slot*" -ExcludeProperty "*Bif";
$slots = #{};
$filteredAttributes.psobject.properties | Foreach { $slots[$_.Name] = $_.Value };
$disabledSlots = $slots.GetEnumerator() |? Value -eq "Disabled";
I am trying to read JSON data which is stored in VM notes. Below is the command i execute to get the VM notes
Get-VM testbox |format-list Notes
The output is
Notes : {
"Program": "AAA",
"Project": "BBBB"
}
I want to read the value of Program into a variable. How can i do it ?
Use ConvertFrom-JSON to parse the JSON-value in your notes-property. I'd store the converted notes in a variable just in case you need to access Project or another part of the json later. Try:
$vm = Get-VM testbox
$notes = $vm.Notes | ConvertFrom-JSON
$mynewvar = $notes.program
Your json is not valid, if it was this would work:
(Get-VM -VMName TestBox).notes | ConvertFrom-Json
Valid Json:
{
"Notes":
{
"Program": "AAA",
"Project": "BBBB"
}
}
Totally untested (and can't test right now) but something like:
Get-VM testbox | Select-Object -ExpandProperty Notes | ConvertFrom-Json
Documentation for ConvertFrom-Json here
Apologies for my lack of powershell knowledge, have been searching far and wide for a solution as i am not much of a programmer.
Background:
I am currently trying to standardise some site settings in Incapsula. To do this i want to maintain a local XML with rules and use some powershell to pull down the existing rules and compare them with what is there to ensure im not doubling up. I am taking this approach of trying to only apply the deltas as:
For most settings incapsula is not smart enough to know it already exists
What can be posted to the API is different varies from what is returned by the API
Examples:
Below is an example of what the API will return on request, this is in a JSON format.
JSON FROM WEBSITE
{
"security": {
"waf": {
"rules": [{
"id": "api.threats.sql_injection",
"exceptions": [{
"values": [{
"urls": [{
"value": "google.com/thisurl",
"pattern": "EQUALS"
}],
"id": "api.rule_exception_type.url",
"name": "URL"
}],
"id": 256354634
}]
}, {
"id": "api.threats.cross_site_scripting",
"action": "api.threats.action.block_request",
"exceptions": [{
"values": [{
"urls": [{
"value": "google.com/anotherurl",
"pattern": "EQUALS"
}],
"id": "api.rule_exception_type.url",
"name": "URL"
}],
"id": 78908790780
}]
}]
}
}
}
And this is the format of the XML with our specific site settings in it
OUR XML RULES
<waf>
<ruleset>
<rule>
<id>api.threats.sql_injection</id>
<exceptions>
<exception>
<type>api.rule_exception_type.url</type>
<url>google.com/thisurl</url>
</exception>
<exception>
<type>api.rule_exception_type.url</type>
<url>google.com/thisanotherurl</url>
</exception>
</exceptions>
</rule>
<rule>
<id>api.threats.cross_site_scripting</id>
<exceptions>
<exception>
<type>api.rule_exception_type.url</type>
<url>google.com/anotherurl</url>
</exception>
<exception>
<type>api.rule_exception_type.url</type>
<url>google.com/anotherurl2</url>
</exception>
</exceptions>
</rule>
</ruleset>
</waf>
I have successfully been able to compare other settings from the site against the XML using the compare-object command, however they had a bit simpler nesting and didn't give me as much trouble. I'm stuck to whether it is a logic problem or a limitation with compare object. An example code is below, it will require the supplied json and xml saved as stack.json/xml in the same directory and should produce the mentioned result :
$existingWaf = Get-Content -Path stack.json | ConvertFrom-Json
[xml]$xmlFile = Get-Content -Path stack.xml
foreach ($rule in $xmlFile)
{
$ruleSet = $rule.waf.ruleset
}
foreach ($siteRule in $ExistingWaf.security.waf.rules)
{
foreach ($xmlRule in $ruleSet)
{
if ($xmlRule.rule.id -eq $siteRule.id)
{
write-output "yes"
$delta = Compare-Object -ReferenceObject #($siteRule.exceptions.values.urls.value | Select-Object) -DifferenceObject #($xmlRule.rule.exceptions.exception.url | Select-Object) -IncludeEqual | where {$xmlRule.rule.id -eq $siteRule.id}
$delta
}
}
}
This is kind of working but not quite what i wanted. I do get a compare between the objects but not for the specific id's, it shows me the results below:
InputObject SideIndicator
----------- -------------
google.com/thisurl ==
google.com/thisanotherurl =>
google.com/anotherurl =>
google.com/anotherurl2 =>
google.com/anotherurl ==
google.com/thisurl =>
google.com/thisanotherurl =>
google.com/anotherurl2 =>
Where as i am more after
InputObject SideIndicator
----------- -------------
google.com/thisurl ==
google.com/thisanotherurl =>
google.com/anotherurl ==
google.com/anotherurl2 =>
Hopefully that makes sense.
Is it possible to only do the compares only on the values where the ids match?
Please let me know if you have any further questions.
Thanks.
The problem was your iteration logic, which mistakenly processed multiple rules from the XML document in a single iteration:
foreach ($xmlRule in $ruleSet) didn't enumerate anything - instead it processed the single <ruleset> element; to enumerate the child <rule> elements, you must use $ruleSet.rule.
$xmlRule.rule.exceptions.exception.url then implicitly iterated over all <rule> children and therefore reported the URLs across all of them, which explains the extra lines in your Compare-Object output.
Here's a streamlined, annotated version of your code:
$existingWaf = Get-Content -LiteralPath stack.json | ConvertFrom-Json
$xmlFile = [xml] (Get-Content -raw -LiteralPath stack.xml )
# No need for a loop; $xmlFile is a single [System.Xml.XmlDocument] instance.
$ruleSet = $xmlFile.waf.ruleset
foreach ($siteRule in $ExistingWaf.security.waf.rules)
{
# !! Note the addition of `.rule`, which ensures that the rules
# !! are enumerated *one by one*.
foreach ($xmlRule in $ruleSet.rule)
{
if ($xmlRule.id -eq $siteRule.id)
{
# !! Note: `$xmlRule` is now a single, rule, therefore:
# `$xmlRule.rule.[...]-> `$xmlRule.[...]`
# Also note that neither #(...) nor Select-Object are needed, and
# the `| where ...` (Where-Object) is not needed.
Compare-Object -ReferenceObject $siteRule.exceptions.values.urls.value `
-DifferenceObject $xmlRule.exceptions.exception.url -IncludeEqual
}
}
}
Additional observations regarding your code:
There is no need to ensure that operands passed to Compare-Object are arrays, so there's no need to wrap them in array sub-expression operator #(...). Compare-Object handles scalar operands fine.
... | Select-Object is a virtual no-op - the input object is passed through[1]
... | Where-Object {$xmlRule.rule.id -eq $siteRule.id} is pointless, because it duplicates the enclosing foreach loop's condition.
Generally speaking, because you're not referencing the pipeline input object at hand via automatic variable $_, your Where-Object filter is static and will either match all input objects (as in your case) or none.
[1] There is a subtle, invisible side effect that typically won't make a difference: Select-Object adds an invisible [psobject] wrapper around the input object, which on rare occasions does cause different behavior later -
see this GitHub issue.