I need some pointers on how to update values in my .json file. Here is a sample fruits.json that I have:
[
{
"ExpireDate": "",
"Origin": "",
"FruitName": "Apple"
},
{
"Expiredate": "",
"Origin": "",
"FruitName": "Orange"
}
]
I have another .JSON file call fruitdetail.json serving as an input file. I need to update the above fruits.json with the info from fruitdetail.json. Basically, lookup the "Fruit Name" from fruitdetail.json and update both "Origin" and "Expire date" keys in fruits.json. Below is what I am starting out with but it doesn't look right.
$fruits = (Get-Content -Path "C:\temp\fruits.json" | ConvertFrom-Json)
$fruitDetail = (Get-Content -Path "C:\temp\fruitdetail.json" | ConvertFrom-Json)
$json = foreach ($fruit in $fruits) {
If ($fruit.FruitName = $fruitDetail.FruitName) {
$Fruit.Origin = $FruitDetail.Origin
$Fruit.ExpireDate = $FruitDetail.ExpireDate
}
}
$json.update
Thanks in advance.
UPDATE: Updated the above code
You're trying to relate data using a common field, similar to an SQL Join. Presumably the fruit name in can be used in this case and for demonstration purposes. Not sure this is perfect, but it should demonstrate the concept.
# Establish Test Date
$JSON_Fruit =
#"
[
{
"Expire Date": "",
"Origin": "",
"Fruit Name": "Apple"
},
{
"Expire date": "",
"Origin": "",
"Fruit Name": "Orange"
}
]
"# |
ConvertFrom-Json
# Establish Test Date
$JSON_FruitDetail = #{}
(#"
[
{
"Expire Date": "10/27/2020",
"Origin": "California",
"Fruit Name": "Apple"
},
{
"Expire date": "11/10/2020",
"Origin": "Arizona",
"Fruit Name": "Orange"
}
]
"# |
ConvertFrom-Json) |
ForEach-Object{
$JSON_FruitDetail.Add( $_.'Fruit Name', $_ )
}
ForEach( $Fruit in $JSON_Fruit )
{
$Fruit."Expire Date" = $JSON_FruitDetail[$Fruit.'Fruit Name']."Expire Date"
$Fruit.Origin = $JSON_FruitDetail[$Fruit.'Fruit Name'].Origin
}
$JSON_Fruit
Note: The parenthesis are to avoid a problem in PowerShell versions prior to 7 (both Windows and Core) In PowerShell 7+ you don't need to use them. For more info see this answer and this one .
Basically you're converting the JSON into PSCustom objects via ConvertFrom-Json. Then storing one side of the data in a dictionary collection keyed on the common field, in this example the "Fruit Name". Then we're looping through the object collection using the "Fruit Name" to reference the entry in the hash table and make the assignment to the destination object's properties.
Note: I've seen a number of PowerShell functions in the community with names like Join-Object that may also be suitable for this, but I usually use techniques like above.
Related
I've to retrieve products and it's details and save it together in a json file. Unfortunately the name and details cannot be retrieved from the same API, I gotta use 2 different API. So I run a code inside a for loop to retrieve the list of product names as well as its details as follows:
For($i=0; $i -lt $cart.total; $i++)
{
$query = 'dummy value with the cart number'
$productName = Invoke-RestMethod -Uri 'https://abc' -Body $query -Method Post
$details = Invoke-RestMethod -Uri 'https://xyz' -Body $query -Method Post
}
All this works, and below is sample data in $productname and $details.
$productName
"iPhone"
$details
"data":[
{"Moduleid": "1", "propertyName": "camera", "value": "12MP"},
{"Moduleid" : "43", "propertyName":"battery", "value": "4000MAj"}
]
However what I am trying to do is to save the $productName and $details into a json as follows:
{
"iPhone 7":[
{"Moduleid": "1", "propertyName": "camera", "value": "12MP"},
{"Moduleid" : "423", "propertyName":"battery", "value": "3000MAh"}
],
"S10+": [
{"Moduleid": "21", "propertyName": "camera", "value": "12MP"},
{"Moduleid" : "43", "propertyName":"battery", "value": "4000MAh"}
]
}
I've tried several ways but couldn't reach any conclusion. Please let me know how can these to variable datas be put into a json format as shown above.
Thank you
The question doesn't make clear what each query is returning. Generally speaking, you need to convert the return JSON from both queries into objects then merge the objects into the objects you really want. The run those objects through ConvertTo-Json which should do the trick. The source object structure is what results in the JSON text etc...
It would look something like below, but again, I have to guess at the starting point, for example I'm sure that $productName isn't a flat array, but is itself JSON.
# For test
$productName = "iPhone","S10+"
# For test
$details =
#"
{
"Data": [{
"Moduleid": "1",
"propertyName": "camera",
"value": "12MP"
},
{
"Moduleid" : "43",
"propertyName": "battery",
"value": "4000MAj"
}]
}
"# |
ConvertFrom-Json
$Results =
For( $i = 0; $i -lt $productName.count; ++$i )
{
[PSCustomObject]#{
$productName[$i] = $details.data[$i]
}
}
$Results | ConvertTo-Json
This would give back:
[
{
"iPhone": {
"Moduleid": "1",
"propertyName": "camera",
"value": "12MP"
}
},
{
"S10+": {
"Moduleid": "43",
"propertyName": "battery",
"value": "4000MAj"
}
}
]
Not quite what you're looking for, but should at least demonstrate the point.
Note: Normally I would look for a common property between the 2
returns to relate the data. Absent that I used the array index. Again
part of the reason this doesn't look right.
If you can please edit the question to include the return JSON from both queries, then we can refine the above.
I created a hashtable like:
$hashTable = [HashTable]::New(0, [StringComparer]::Ordinal)
This is so that the keys are case sensitive. Normally if you create it as #{}, it won't be case sensitive.
Then i mapped the key and value to the hash table as follows,
$hashTable.Add($productName.data.name,$details.data)
Now, I convert it to Json as I wanted it using ConvertTo-Json
$hashTable| ConvertTo-Json | Out-File $outputPath
Here is a simplified json file of a terraform state file (let's call it dev.ftstate)
{
"version": 4,
"terraform_version": "0.12.9",
"serial": 2,
"lineage": "ba56cc3e-71fd-1488-e6fb-3136f4630e70",
"outputs": {},
"resources": [
{
"module": "module.rds.module.reports_cpu_warning",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds.module.reports_lag_warning",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds.module.cross_region_replica_lag_alert",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds",
"mode": "managed",
"type": "aws_db_instance",
"name": "master",
"provider": "provider.aws",
"instances": [
{
"schema_version": 0,
"attributes": {
"address": "dev-database.123456.us-east-8.rds.amazonaws.com",
"allocated_storage": 10,
"password": "",
"performance_insights_enabled": false,
"tags": {
"env": "development"
},
"timeouts": {
"create": "6h",
"delete": "6h",
"update": "6h"
},
"timezone": "",
"username": "admin",
"vpc_security_group_ids": [
"sg-1234"
]
},
"private": ""
}
]
}
]
}
There are many modules at the same level of module.rds inside the instances. I took out many of them to create the simplified version of the raw data. The key takeway: do not assume the array index will be constant in all cases.
I wanted to extract the password field in the above example.
My first attempt is to use equality check to extract the relevant modules
` jq '.resources[].module == "module.rds"' dev.tfstate`
but it actually just produced a list of boolean values. I don't see any mention of builtin functions like filter in jq's manual
then I tried to just access the field:
> jq '.resources[].module[].attributes[].password?' dev.tfstate
then it throws the following error
jq: error (at dev.tfstate:1116): Cannot iterate over string ("module.rds")
So what is the best way to extract the value? Hopefully it can only focus on the password attribute in module.rds module only.
Edit:
My purpose is to detect if a password is left inside a state file. I want to ensure the passwords are exclusively stored in AWS secret manager.
You can extract the module you want like this.
jq '.resources[] | select(.module == "module.rds")'
I'm not confident that I understand the requirements for the rest of the solution. So this might not only not be the best way of doing what you want; it might not do what you want at all!
If you know where password will be, you can do this.
jq '.resources[] | select(.module == "module.rds") | .instances[].attributes.password'
If you don't know exactly where password will be, this is a way of finding it.
jq '.resources[] | select(.module == "module.rds") | .. | .password? | values'
According to the manual under the heading "Recursive Descent," ..|.a? will "find all the values of object keys “a” in any object found “below” ."
values filters out the null results.
You could also get the password value out of the state file without jq by using Terraform outputs. Your module should define an output with the value you want to output and you should also output this at the root module.
Without seeing your Terraform code you'd want something like this:
modules/rds/main.tf
resource "aws_db_instance" "master" {
# ...
}
output "password" {
value = aws_db_instance.master.password
sensitive = true
}
example/main.tf
module "rds" {
source = "../modules/rds"
# ...
}
output "rds_password" {
value = module.rds.password
sensitive = true
}
The sensitive = true parameter means that Terraform won't print the output to stdout when running terraform apply but it's still held in plain text in the state file.
To then access this value without jq you can use the terraform output command which will retrieve the output from the state file and print it to stdout. From there you can use it however you want.
I am pulling data from an API using the New-WebServiceProxy in PowerShell 4.0 and then piping it out to a JSON file for review and import on another API service (same API version, etc, just a different host).
$tasklist.Taskconfig | ConvertTo-JSON-Depth 50 -As String | Out-File -FilePath $exportpath\$name.xml -Force
Gives me my XML containing the TaskConfig. In this case, TaskConfig is an object type automatically generated by the API I'm interfacing with. When I want to import the content I am using:
$taskconfig = (Get-Content "$taskjson") -join "`n" | ConvertFrom-Json
but when I run this it's unable to create the object. I assume this is because the JSON contains nested children, giving the error-
Cannot convert value "#{Name=plugindive; Value=;> Children=System.Object[]}" to type "Microsoft.PowerShell.Commands.NewWebserviceProxy.AutogeneratedTypes.WebServiceProxy1rcleWeb_WebClientAPI_asmx_wsdl.TaskConfig". Error: "Cannot convert the "#{Name=plugindive; Value=;Children=System.Object[]}" value of type "System.Management.Automation.PSCustomObject" to type "Microsoft.PowerShell.Commands.NewWebserviceProxy.AutogeneratedTypes.WebServiceProxy1rcleWeb_WebClientAPI_asmx_wsdl.TaskConfig"."
I've tried explictly stating the type of object:
$taskconfig = [Microsoft.PowerShell.Commands.NewWebserviceProxy.AutogeneratedTypes.WebServiceProxy1rcleWeb_WebClientAPI_asmx_wsdl.TaskConfig](Get-Content "$taskjson" | Out-string | ConvertFrom-Json)
as well as creating the object then trying to add the children from my JSON -
$taskconfig.children = $json.children
But these all fail in the same way.
I don't seem to get this same issue in PowerShell 5.0 interestingly enough, but I can't verify why - is there another way to approach this?
Added example JSON below
{"Name": "plugindive",
"Value": null,
"Children": [{
"Name": "auto",
"Value": "False",
"Children": [
]
},
{
"Name": "categories",
"Value": null,
"Children": [{
"Name": "Module Z",
"Value": "False",
"Children": [
]
},
{
"Name": "Module A",
"Value": "False",
"Children": [
]
},
{
"Name": "Module B",
"Value": "False",
"Children": [
]
},
{
"Name": "Module C",
"Value": "False",
"Children": [
]
}
]
}
]
}
It seems as if this doesn't work in PowerShell v3.0, so I simply ended up making posts with the explicit XML directly, rather than converting to JSON.
I have a messages.json file
[
{
"id": "title",
"description": "This is the Title",
"defaultMessage": "title",
"filepath": "src/title.js"
},
{
"id": "title1",
"description": "This is the Title1",
"defaultMessage": "title1",
"filepath": "src/title1.js"
},
{
"id": "title2",
"description": "This is the Title2",
"defaultMessage": "title2",
"filepath": "src/title2.js"
},
{
"id": "title2",
"description": "This is the Title2",
"defaultMessage": "title2",
"filepath": "src/title2.js"
},
]
I want to create an object
{
"title": "Dummy1",
"title1": "Dummy2",
"title2": "Dummy3",
"title3": "Dummy4"
}
from the top one.
So far I have
jq '.[] | .id' src/messages.json;
And it does give me the IDs
How do I add some random text and make the new object as above?
Can we also create a new JSON file and write the newly created object onto it using jq?
Your output included "title3" so I'll assume that you intended that the second occurrence of "title2" in the input was supposed to refer to "title3".
With this assumption, the following jq program seems to do what you want:
map( .id )
| . as $in
| reduce range(0;length) as $i ({};
. + {($in[$i]): "dummy\(1+$i)"})
In words, extract the values of .id, and then turn each into an object of the form: {(.id) : "dummy\(1+$i)"}
This uses string interpolation, and produces:
{
"title": "dummy1",
"title1": "dummy2",
"title2": "dummy3",
"title3": "dummy4"
}
reduce-free solution
map(.id )
| [., [range(0;length)]]
| transpose
| map( {(.[0]): "dummy\(.[1]+1)"})
| add
Output
Can we also create a new json file and write the newly created object onto it using jq?
Yes, just use output redirection:
jq -f program.jq messages.json > output.json
Addendum
I want a parent object "de" to the already created json file objects
You could just pipe either of the above solutions to: {de: .}
I've got a jq question. Given a file file.json containing:
[
{
"type": "A",
"name": "name 1",
"url": "http://domain.com/path/to/filenameA.zip"
},
{
"type": "B",
"name": "name 2",
"url": "http://domain.com/otherpath/to/filenameB.zip"
},
{
"type": "C",
"name": "name 3",
"url": "http://otherdomain.com/otherpath/to/filenameB.zip"
}
]
I'm looking to create another file using jq with url modified only if the url's value matches some pattern. For example, I'd want to update any url matching the pattern:
http://otherdomain.com.*filenameB.*
to some fixed string such as:
http://yetanotherdomain.com/new/path/to/filenameC.tar.gz
with the resulting json:
[
{
"type": "A",
"name": "name 1",
"url": "http://domain.com/path/to/filenameA.zip"
},
{
"type": "B",
"name": "name 2",
"url": "http://domain.com/otherpath/to/filenameB.zip"
},
{
"type": "C",
"name": "name 3",
"url": "http://yetanotherdomain.com/new/path/to/filenameB.tar.gz"
}
]
I haven't gotten far even on being able to find the url, let alone update it. This is as far as I've gotten (wrong results and doesn't help me with the update issue):
% cat file.json | jq -r '.[] | select(.url | index("filenameB")).url'
http://domain.com/otherpath/to/filenameB.zip
http://otherdomain.com/otherpath/to/filenameB.zip
%
Any ideas on how to get the path of the key that has a value matching a regex? And after that, how to update the key with some new string value? If there are multiple matches, all should be updated with the same new value.
The good news is that there's a simple solution to the problem:
map( if .url | test("http://otherdomain.com.*filenameB.*")
then .url |= sub( "http://otherdomain.com.*filenameB.*";
"http://yetanotherdomain.com/new/path/to/filenameC.tar.gz")
else .
end)
The not-so-good news is that it's not so easy to explain unless you understand the key cleverness here - the "|=" filter. There is plenty of jq documentation about it, so I'll just point out that it is similar to the += family of operators in the C family of programming languages.
Specifically, .url |= sub(A;B) is like .url = (.url|sub(A;B)). That is how the update is done "in-place".
Here is a solution which identifies paths to url members with tostream and select and then updates the values using reduce and setpath
"http://otherdomain.com.*filenameB.*" as $from
| "http://yetanotherdomain.com/new/path/to/filenameC.tar.gz" as $to
| reduce (tostream | select(length == 2 and .[0][-1] == "url")) as $p (
.
; setpath($p[0]; $p[1] | sub($from; $to))
)