read hyperopt parameters from json - json

I want to read hyperopt parameters from a JSON file.
My JSON file would be like:
[
{
"id": "121",
"model": [
{
"model_name": "power",
"estimator_type": [
{
"type": "Polynomial",
"degree": [2, 3, 4]
},
{
"type": "svm",
"C": [0, 1],
"kernel": [
{
"ktype": "linear"
},
{
"ktype": "RBF",
"width": [0, 1]
}
]
}
],
"cut_values": {
"qids": ["1234"]
}
},
{
"model_name": "speed",
"estimator_type": [
{
"type": "Polynomial",
"degree": ["quniform", 2, 3]
}
],
"cut_values": null
}
]
},
{
"id": "123",
"model": [
{
"model_name": "power",
"estimator_type": [
{
"type": "LinearRegression"
}
],
"cut_values": null
}
]
}
]
I have checked this post but with no success for more complex JSON like the one above.
I want to be able to create a space like 2.2 A Search Space Example: scikit-learn.

Related

How can I create a hierarchical json response in FLASK

I have a single table in database like database table. I want to search a child from database and return a hierarchical JSON to a front end in order to create a tree. How can I do that in FLASK.
My expected JSON for mat should be like expected JSON

Since you have tagged your question with flask, this post assumes you are using Python as well. To format your database values in JSON string, you can query the db and then use recursion:
import sqlite3, collections
d = list(sqlite3.connect('file.db').cursor().execute("select * from values"))
def get_tree(vals):
_d = collections.defaultdict(list)
for a, *b in vals:
_d[a].append(b)
return [{'name':a, **({} if not (c:=list(filter(None, b))) else {'children':get_tree(b)})} for a, b in _d.items()]
import json
print(json.dumps(get_tree(d), indent=4))
Output:
[
{
"name": "AA",
"children": [
{
"name": "BB",
"children": [
{
"name": "EE",
"children": [
{
"name": "JJ",
"children": [
{
"name": "EEV"
},
{
"name": "FFW"
}
]
},
{
"name": "KK",
"children": [
{
"name": "HHX"
}
]
}
]
}
]
},
{
"name": "CC",
"children": [
{
"name": "FF",
"children": [
{
"name": "LL",
"children": [
{
"name": "QQY"
}
]
},
{
"name": "MM",
"children": [
{
"name": "RRV"
}
]
}
]
},
{
"name": "GG",
"children": [
{
"name": "NN",
"children": [
{
"name": "SSW"
}
]
}
]
}
]
},
{
"name": "DD",
"children": [
{
"name": "HH",
"children": [
{
"name": "OO",
"children": [
{
"name": "TTZ"
}
]
}
]
},
{
"name": "II",
"children": [
{
"name": "PP",
"children": [
{
"name": "UUW"
}
]
}
]
}
]
}
]
}
]

Merge objects in same array on single key

I have an array of JSON objects formatted as follows:
[
{
"id": 1,
"names": [
{
"name": "Bulbasaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
},
{
"id": 1,
"types": [
{
"slot": 1,
"type": {
"name": "grass",
"url": "http://myserver.com:8000/api/v2/type/12/"
}
},
{
"slot": 2,
"type": {
"name": "poison",
"url": "http://myserver.com:8000/api/v2/type/4/"
}
}
]
},
{
"id": 2,
"names": [
{
"name": "Ivysaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
},
{
"id": 2,
"types": [
{
"slot": 1,
"type": {
"name": "ice",
"url": "http://myserver.com:8000/api/v2/type/10/"
}
},
{
"slot": 2,
"type": {
"name": "electric",
"url": "http://myserver.com:8000/api/v2/type/8/"
}
}
]
},
{
"id": 3,
"names": [
{
"name": "Venusaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
},
{
"id": 3,
"types": [
{
"slot": 1,
"type": {
"name": "ground",
"url": "http://myserver.com:8000/api/v2/type/2/"
}
},
{
"slot": 2,
"type": {
"name": "rock",
"url": "http://myserver.com:8000/api/v2/type/3/"
}
}
]
}
]
Note that these are pairs of separate objects that appear sequentially in a JSON array, with each pair sharing an id field. This pattern repeats several hundred times in the array. What I need to accomplish is to "merge" each id-sharing pair into one object. So, the resultant output would be
[
{
"id": 1,
"names": [
{
"name": "Bulbasaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
"types": [
{
"slot": 1,
"type": {
"name": "grass",
"url": "http://myserver.com:8000/api/v2/type/12/"
}
},
{
"slot": 2,
"type": {
"name": "poison",
"url": "http://myserver.com:8000/api/v2/type/4/"
}
}
]
},
{
"id": 2,
"names": [
{
"name": "Ivysaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
"types": [
{
"slot": 1,
"type": {
"name": "ice",
"url": "http://myserver.com:8000/api/v2/type/10/"
}
},
{
"slot": 2,
"type": {
"name": "electric",
"url": "http://myserver.com:8000/api/v2/type/8/"
}
}
]
},
{
"id": 3,
"names": [
{
"name": "Venusaur",
"language": {
"name": "en",
"url": "http://myserver.com:8000/api/v2/language/9/"
}
},
],
"types": [
{
"slot": 1,
"type": {
"name": "ground",
"url": "http://myserver.com:8000/api/v2/type/2/"
}
},
{
"slot": 2,
"type": {
"name": "rock",
"url": "http://myserver.com:8000/api/v2/type/3/"
}
}
]
}
]
I've gotten these objects to appear next to each other via the group_by(.id) command, but I'm at a loss as to how I should actually combine them. I'm very much still a novice with jq so I'm a bit overwhelmed with the amount of possible solutions.

[Note: The following assumes that the data shown in the Q have been corrected so that they are valid JSON.]
The merging you want can be achieved by object addition (x + y). For example, given the two JSON objects as shown in the question (i.e., as a stream), you could write:
jq -s '.[0] + .[1]'
However, since the question also indicates these objects are actually in an array, let's next consider the case of an array with two objects. In that case, you could simply write:
jq add
Finally, if you have an array of arrays each of which is an array of objects, you could use map(add). Since you don't have a very large array, you could simply write:
group_by(.id) | map(add)
Please note that jq defines object addition in a non-commutative way. Specifically, there is a bias towards the right-most key.

Recurse for object if exists in JQ

I have the following structure:
{
"hits":
[
{
"_index": "main"
},
{
"_index": "main",
"accordions": [
{
"id": "1",
"accordionBody": "body1",
"accordionInnerButtonTexts": [
"button11",
"button12"
]
},
{
"id": "2",
"accordionBody": "body2",
"accordionInnerButtonTexts": [
"button21",
"button22"
]
}
]
}
]
}
I want to get to this structure:
{
"index": "main"
}
{
"index": "main",
"accordions":
[
{
"id": "1",
"accordionBody": "body1",
"accordionInnerButtonTexts": [
"button11",
"button12"
]
},
{
"id": "2",
"accordionBody": "body2",
"accordionInnerButtonTexts": [
"button21",
"button22"
]
}
]
}
Which means that I always want to include the _index-field as index, and I want to include the whole accordions-list IF IT EXISTS in the object. Here is my attempt:
.hits[] | {index: ._index, accordions: recurse(.accordions[]?)}
It does not produce what I want:
{
"index": "main",
"accordions": {
"_index": "main"
}
}
{
"index": "main",
"accordions": {
"_index": "main",
"accordions": [
{
"id": "1",
"accordionBody": "body1",
"accordionInnerButtonTexts": [
"button11",
"button12"
]
},
{
"id": "2",
"accordionBody": "body2",
"accordionInnerButtonTexts": [
"button21",
"button22"
]
}
]
}
}
{
"index": "main",
"accordions": {
"id": "1",
"accordionBody": "body1",
"accordionInnerButtonTexts": [
"button11",
"button12"
]
}
}
{
"index": "main",
"accordions": {
"id": "2",
"accordionBody": "body2",
"accordionInnerButtonTexts": [
"button21",
"button22"
]
}
}
It seems to create a list of all different permutations given by mixing the objects. This is not what I want. What is the correct jq command, and what is my mistake?

The problem as stated does not require any recursion. Using your attempt as a model, one could in fact simply write:
.hits[]
| {index: ._index}
+ (if has("accordions") then {accordions} else {} end)
Or, with quite different semantics:
.hits[] | {index: ._index} + . | del(._index)

Aggregate and sum json data in python

I am new to python, using python3. I have json data like:
{
"message": {
"count": 46,
"limit": 1000,
"schools": [
{
"class": "1",
"class_id": "1c8***",
"charges": [
{
"cost": 10,
"breakdown": [
{
"books": "1",
"unitQuantity": "10"
}
]
}
],
"area": "maccau"
},
{
"class": "2",
"class_id": "1c3***",
"charges": [
{
"cost": 100,
"breakdown": [
{
"books": "1",
"unitQuantity": "100"
}
]
}
],
"area": "maccau"
},
{
"class": "1",
"class_id": "1c3***",
"charges": [
{
"cost": 10,
"breakdown": [
{
"books": "1",
"unitQuantity": "10"
}
]
}
],
"area": "maccau"
},
{
"class": "2",
"class_id": "1c8***",
"charges": [
{
"cost": 50,
"breakdown": [
{
"books": "1",
"unitQuantity": "50"
}
]
}
],
"area": "maccau"
}
],
"url": {
"link": "/"
}
}
}
I was able to use json.loads to load data and I am trying to get results like:
class Cost
1 20
2 150
I tried converting json to a dictionary:
item_dict = json.load(json_data)
Tried to get data out using for loop and checking if class = 1 and then summing up the cost. But I feel like that is not the best approach. Can someone please tell me what would be the best way of doing this?

Update JSON file using PowerShell

I am currently trying to setup a continuous integration system using VSTS and have run into a bit of a snag. As part of the release process I need to update a specific object value in a JSON file depending on the environment. The only tools it seems I have at my disposal that might get this done in the VSTS environment is PowerShell.
I've done quite a bit of research and have not been able to figure out how exactly this can be done. I found this question and answer here on Stack Overflow "How do I update JSON file using PowerShell" but executing the script provided in the answer changes the structure of the JSON file substantially and adds quite a bit of what looks like PowerShell metadata.
Ideally, I would like to take an existing JSON file that gets deployed and update the value of the connectionString property in the example JSON below.
{
"policies": {
"Framework.DataContext": {
"connectionString": "Server=ServerName;Database=DateBaseName;Integrated Security=sspi;"
}
}
}
Does anyone have any advice on how to accomplish this? So far I have tried running the following script but it throws an "The property 'connectionString' cannot be found on this object. Verify that the property exists and can be set." exception. I have verified that the object traversal is correct and the connectionString property exists.
$pathToJson = "D:\Path\To\JSON\file.json"
$a = Get-Content $pathToJson | ConvertFrom-Json
$a.policies.'Framework.DataContext'.connectionString = "Server=ServerName;Database=DateBaseName;Integrated Security=sspi;"
$a | ConvertTo-Json | set-content $pathToJson
The full contents of file.json are as follows
{
"log": {
"level": 0,
"file": "c:\\temp\\simport.log",
"formats": {
"error": null,
"start": null,
"requestBegin": null,
"requestWork": "",
"requestError": null,
"requestEnd": null,
"stop": null
},
"eventLog": {
"name": "Application"
}
},
"diagnostic": {
"stackTrace": false
},
"api": {
"simport": true
},
"roles": {
"0": "Anonymous",
"1": "Administrator",
"2": "Participant",
"3": "Facilitator"
},
"pathType": {
"area": 1,
"region": 2,
"session": 3,
"team": 4
},
"scenarios": {
"default": {
"default": true,
"initState": "Init",
"rounds": [
{
"name": "round1",
"displayName": "R1",
"beginTime": 1,
"endTime": 3
},
{
"name": "round2",
"displayName": "R2",
"beginTime": 4,
"endTime": 6
},
{
"name": "round3",
"displayName": "R3",
"beginTime": 7,
"endTime": 9
},
{
"name": "round4",
"displayName": "R4",
"beginTime": 10,
"endTime": 12
}
]
}
},
"simportQueries": {
"package": "bin/trc.simport3.zip"
},
"customQueries": {
"package": "app/config/custom-queries.zip",
"parameters": {
}
},
"audit": {
"Path.Create": true,
"Path.Delete": true,
"Team.Create": true,
"Team.Update": true,
"Team.Delete": true,
"SimportData.SaveValues": true
},
"tasks": {
"task1": {
"state": "",
"required": "",
"completed": "C:Task1Status:+0"
}
},
"feedback": {
"welcome": {
"text": {
"": "en-us",
"en-us": "Welcome"
}
}
},
"contentCategories": {
"demo1": {
"round": 1
}
},
"policies": {
"Simport.Web.Module": {
"fileMask": ".aspx,.asmx",
"deny": {
"statusCode": 404,
"statusDescription": "Not found",
"location": [
"/{0,1}app/config/(.*\\.json)$",
"/{0,1}app/config/(.*\\.xml)$",
"/{0,1}app/config/(.*\\.zip)$",
"/{0,1}app/config/(.*\\.xlsx)$"
]
},
"formDataContentType": [ "application/x-www-form-urlencoded" ]
},
"Framework.DataContext": {
"connectionString": "Server=(local);Database=Simport3;Integrated Security=sspi;",
"commandTimeout": 30
},
"Simport.Security": {
"passwordEncryption": "",
"passwordSalt": "",
"passwordPolicy": {
"disabled": true,
"min": 8,
"max": 100,
"rules": [
{ "id": "digit", "pattern": "\\d+", "flags": "i" },
{ "id": "letter", "pattern": "\\w+", "flags": "i" },
{ "id": "upper", "pattern": "[A-Z]+" },
{ "id": "lower", "pattern": "[a-z]+" },
{ "id": "special", "pattern": "[\\!##\\$_~]+", "flags": "i" },
{ "id": "prohibited", "pattern": "[\\\\/'\"\\?\\^&\\+\\-\\*\\%\\:;,\\.]+", "flags": "gi", "match": false }
]
}
},
"Simport.PackageDefinition": {
"path": "~/app/config/manifest.xml"
},
"Security.SignIn": {
"result": {
"default": "u,p,p.props,t"
},
"claims": [
[ "userId", "firstName", "lastName" ]
]
},
"Security.GetContext": {
"result": {
"default": "u,p,p.props,pr,t"
}
},
"Security.ChangePassword": {
"allowedRoles": [ 1, 2, 3 ]
},
"Security.ResetPassword": {
"allowedRoles": [ 1, 2 ]
},
"Security.Register": {
"allowedRoles": [ 0 ],
"!pathType-0": 4,
"!roleId-0": 2
},
"Path.Create": {
"allowedRoles": [ 1, 2, 3 ]
},
"Path.Select": {
"allowedRoles": [ 1, 2, 3 ],
"result-1": {
}
},
"Path.Delete": {
"allowedRoles": [ 1, 2 ]
},
"User.Select": {
"allowedRoles": [ 1, 2, 3 ],
"result": {
"select": [ "id", "pathid", "roleid", "name", "email", "login", "props" ],
"restrict": [ "password" ]
},
"result-1": {
"select": "*",
"group": true
}
},
"User.Create": {
"allowedRoles": [ 1, 2 ],
"result": {
"select": [ "id", "pathid", "roleid", "name", "email", "login", "props" ],
"restrict": [ "password" ]
},
"result-1": {
"select": "*",
"group": true
}
},
"User.Update": {
"allowedRoles": [ 1, 2, 3 ],
"result": {
"select": [ "id", "pathid", "roleid", "name", "email", "login", "props" ],
"restrict": [ "password" ]
}
},
"User.Delete": {
"allowedRoles": [ 1, 2 ],
"result": {
"restrict": [ "password" ]
}
},
"Session.Select": {
"allowedRoles": [ 1, 2, 3 ],
"enforcePathLevel": true,
"result": {
"default": [ "name", "beginDate", "endDate" ],
"restrict": [ "password" ]
},
"result-1": {
"default": [ "name", "beginDate", "endDate" ],
"treeAllowed": true,
"treeDefault": false
}
},
"Session.Create": {
"allowedRoles": [ 1, 2 ],
"enforcePathLevel": true
},
"Session.Update": {
"allowedRoles": [ 1, 2 ],
"enforcePathLevel": true,
"update-restictions": [ "password" ],
"update-restictions-1": [ ],
"result": {
"restrict": [ "password" ]
}
},
"Session.Delete": {
"allowedRoles": [ 1, 2 ],
"result": {
"restrict": [ "password" ]
}
},
"Team.Select": {
"allowedRoles": [ 1, 2, 3 ],
"enforcePathLevel": false,
"enforcePathLevel-1": true,
"result-1": {
"treeAllowed": true,
"treeDefault": false
}
},
"Team.Create": {
"allowedRoles": [ 1, 2, 3 ],
"enforcePathLevel": true,
"enforcePathLevel-1": false,
"allowMultiple": false,
"allowMultiple-1": true,
"result": {
},
"overrides": {
"roleID": 3
}
},
"Team.Reset": {
"allowedRoles": [ 1, 2, 3 ]
},
"Team.Delete": {
"allowedRoles": [ 1, 2, 3 ],
"deleteMultiple": true,
"result": {
"default": "t"
}
},
"Team.TransitionTo": {
"allowedRoles": [ 1, 2, 3 ],
"inboundRules": {
"Round1Init": {
"allowedRoles": [ ]
}
},
"outboundRules": {
"Round1Wait": {
"allowedRoles": [ 1, 2, 3 ]
}
}
},
"Team.TakeControl": {
"allowedRoles": [ 1, 2, 3, 4 ],
"select-1": {
"select": "*",
"restrict": [ "ctrl.userID", "ctrl.loginName" ]
}
},
"Team.ReleaseControl": {
"allowedRoles": [ 1, 2, 3, 4 ],
"select-1": {
"select": "*",
"restrict": [ "ctrl.userID", "ctrl.loginName" ]
}
},
"Team.GetStatus": {
"allowedRoles": [ 1, 2, 3 ],
"result": {
"default": "p,t,pr"
}
},
"Data.Select": {
"allowedRoles": [ 1, 2, 3 ]
},
"Data.Update": {
"allowedRoles": [ 1, 2, 3 ],
"audit": {
"g": false,
"i": true,
"o": true,
"s": true
}
},
"Data.ExecuteQuery": {
"allowedRoles": [ 0, 1, 2, 3 ],
"allowed-0": [ "login4\\areas", "login4\\regions", "login4\\sessions", "login4\\teams" ],
"restrict-3": [ "prohibitedQueryNameHere" ]
},
"Document.Select": {
"defaultTextEncoding": "utf-16"
},
"Document.Create": {
"~allowFileExt": [ ],
"denyFileExt": [ ".exe", ".com", ".cmd", ".bat", ".ps1" ],
"~allowContentType": [ ],
"denyContentType": [ "application/x-msdownload" ],
"maxContentLength": 0,
"defaultTextEncoding": "utf-16"
},
"Document.Update": {
"allowedRoles": [ 1, 2, 3 ]
},
"Document.Delete": {
},
"Document.Download": {
}
}
}

Your json is missing the starting and ending curly brackets:
{
"policies": {
"Framework.DataContext": {
"connectionString": "Server=ServerName;Database=DateBaseName;Integrated Security=sspi;"
}
}
}
Now you can update the file like this:
$pathToJson = "F:\Path\To\JSON\file.json"
$a = Get-Content $pathToJson | ConvertFrom-Json
$a.policies.'Framework.DataContext'.connectionString = "Server=ServerName;Database=DateBaseName;Integrated Security=sspi2;"
$a | ConvertTo-Json | set-content $pathToJson
You could also use some Select-Object to get the property:
$connectionString = $a | select -expand policies | select -expand Framework.DataContext
$connectionString.connectionString = 'Test'

$s = Get-Content "F:\Path\To\JSON\file.json" -Raw|ConvertFrom-Json
$s.policies.'Framework.DataContext'.connectionString="Server=ServerName;Database=DateBaseName;Integrated Security=sspi2;"
$s|ConvertTo-Json |Set-Content "F:\Path\To\JSON\file.json"

I also faced the similar problem. Fixed it by specifying the INDEX of the object I was trying to edit.
$a.policies[0].'Framework.DataContext'.connectionString = "Server=ServerName;Database=DateBaseName;Integrated Security=sspi;"
Hope it helps!