Json dynamic fields to struct issue [duplicate] - json

This question already has answers here:
Decoding generic JSON objects to one of many formats
(1 answer)
How to parse a complicated JSON with Go unmarshal?
(3 answers)
golang | unmarshalling arbitrary data
(2 answers)
How to parse JSON in golang without unmarshaling twice
(3 answers)
Is it possible to partially decode and update JSON? (go)
(2 answers)
Closed 10 months ago.
Every time I resolve one issue I get into another similar but different.
I need to convert into struts below JSON, but some parts are dynamic. and I am not very experienced with this conversions and I get very confused.
Below I have two fields that are different not static ( I was able with the help of this forum resolve a similar issue with one dynamic entry, but now I have two).
the fields I called them "This string changes" so is less confusing.
[null,null,"hub:zWXKROOM","presence_state",{"74ce1906-af89-48a9-aec7-501369509000":{"metas":[{"context":{"embed":false,"mobile":false},"permissions":{"close_hub":false,"embed_hub":false,"fly":true,"join_hub":true,"kick_users":false,"mute_users":false,"pin_objects":false,"spawn_and_move_media":true,"spawn_camera":true,"spawn_drawing":false,"spawn_emoji":true,"update_hub":false,"update_hub_promotion":false,"update_roles":false},"phx_ref":"tGJf9IxredI=","phx_ref_prev":"zJ3pFzeYafM=","presence":"room","profile":{"avatarId":"3IADk9x","displayName":"real changeling"},"roles":{"creator":false,"owner":false,"signed_in":false}}]},"774e91d5-a324-47d7-ba75-edf9ed5bbe1a":{"metas":[{"context":{"embed":false,"mobile":false},"permissions":{"close_hub":false,"embed_hub":false,"fly":true,"join_hub":true,"kick_users":false,"mute_users":false,"pin_objects":false,"spawn_and_move_media":true,"spawn_camera":true,"spawn_drawing":false,"spawn_emoji":true,"update_hub":false,"update_hub_promotion":false,"update_roles":false},"phx_ref":"NnAT0YpIaUg=","phx_ref_prev":"tzv+xV6h0Rs=","presence":"room","profile":{"avatarId":"PcJ8Sxb","displayName":"GoBotWebSockets"},"roles":{"creator":false,"owner":false,"signed_in":false}}]},"9bd22f70-521a-49c2-9cb9-ac58dabfa1d6":{"metas":[{"context":{"embed":false,"mobile":false},"permissions":{"close_hub":false,"embed_hub":false,"fly":true,"join_hub":true,"kick_users":false,"mute_users":false,"pin_objects":false,"spawn_and_move_media":true,"spawn_camera":true,"spawn_drawing":false,"spawn_emoji":true,"update_hub":false,"update_hub_promotion":false,"update_roles":false},"phx_ref":"kPknFXlNkMo=","phx_ref_prev":"IA2Es263VdA=","presence":"room","profile":{"avatarId":"3IADk9x","displayName":"killab33z"},"roles":{"creator":false,"owner":false,"signed_in":false}}]},"f87b718a-c873-40a9-99db-91b4d0f7f4de":{"metas":[{"context":{"embed":false,"mobile":false},"permissions":{"close_hub":true,"embed_hub":true,"fly":true,"join_hub":true,"kick_users":true,"mute_users":true,"pin_objects":true,"spawn_and_move_media":true,"spawn_camera":true,"spawn_drawing":true,"spawn_emoji":true,"update_hub":true,"update_hub_promotion":false,"update_roles":true},"phx_ref":"XCCt44iesAo=","presence":"lobby","profile":{"avatarId":"https://s3.amazonaws.com/readyplayerbaker/avatars_baked/89e86e1a-43c7-4520-8f91-9a94ed42a722.glb","displayName":"ReK2"},"roles":{"creator":true,"owner":true,"signed_in":true}}]}}]
with a lot of help of someone here I was able to convert a very similar one but with two I cant seem to get it to work. I keep getting empty results..
the similar approach is Here
I have tried replicating with a field above and other similar things.
I have look many blogs for this situation, no luck.
I also have try this tools, but is not really working because the tool does not know the details
edit: here is from json lint:
[
null,
null,
"hub:zWXKROOM",
"presence_state",
{
"74ce1906-af89-48a9-aec7-501369509000": {
"metas": [
{
"context": {
"embed": false,
"mobile": false
},
"permissions": {
"close_hub": false,
"embed_hub": false,
"fly": true,
"join_hub": true,
"kick_users": false,
"mute_users": false,
"pin_objects": false,
"spawn_and_move_media": true,
"spawn_camera": true,
"spawn_drawing": false,
"spawn_emoji": true,
"update_hub": false,
"update_hub_promotion": false,
"update_roles": false
},
"phx_ref": "tGJf9IxredI=",
"phx_ref_prev": "zJ3pFzeYafM=",
"presence": "room",
"profile": {
"avatarId": "3IADk9x",
"displayName": "real changeling"
},
"roles": {
"creator": false,
"owner": false,
"signed_in": false
}
}
]
},
"774e91d5-a324-47d7-ba75-edf9ed5bbe1a": {
"metas": [
{
"context": {
"embed": false,
"mobile": false
},
"permissions": {
"close_hub": false,
"embed_hub": false,
"fly": true,
"join_hub": true,
"kick_users": false,
"mute_users": false,
"pin_objects": false,
"spawn_and_move_media": true,
"spawn_camera": true,
"spawn_drawing": false,
"spawn_emoji": true,
"update_hub": false,
"update_hub_promotion": false,
"update_roles": false
},
"phx_ref": "NnAT0YpIaUg=",
"phx_ref_prev": "tzv+xV6h0Rs=",
"presence": "room",
"profile": {
"avatarId": "PcJ8Sxb",
"displayName": "GoBotWebSockets"
},
"roles": {
"creator": false,
"owner": false,
"signed_in": false
}
}
]
},
"9bd22f70-521a-49c2-9cb9-ac58dabfa1d6": {
"metas": [
{
"context": {
"embed": false,
"mobile": false
},
"permissions": {
"close_hub": false,
"embed_hub": false,
"fly": true,
"join_hub": true,
"kick_users": false,
"mute_users": false,
"pin_objects": false,
"spawn_and_move_media": true,
"spawn_camera": true,
"spawn_drawing": false,
"spawn_emoji": true,
"update_hub": false,
"update_hub_promotion": false,
"update_roles": false
},
"phx_ref": "kPknFXlNkMo=",
"phx_ref_prev": "IA2Es263VdA=",
"presence": "room",
"profile": {
"avatarId": "3IADk9x",
"displayName": "killab33z"
},
"roles": {
"creator": false,
"owner": false,
"signed_in": false
}
}
]
},
"f87b718a-c873-40a9-99db-91b4d0f7f4de": {
"metas": [
{
"context": {
"embed": false,
"mobile": false
},
"permissions": {
"close_hub": true,
"embed_hub": true,
"fly": true,
"join_hub": true,
"kick_users": true,
"mute_users": true,
"pin_objects": true,
"spawn_and_move_media": true,
"spawn_camera": true,
"spawn_drawing": true,
"spawn_emoji": true,
"update_hub": true,
"update_hub_promotion": false,
"update_roles": true
},
"phx_ref": "XCCt44iesAo=",
"presence": "lobby",
"profile": {
"avatarId": "https://s3.amazonaws.com/readyplayerbaker/avatars_baked/89e86e1a-43c7-4520-8f91-9a94ed42a722.glb",
"displayName": "ReK2"
},
"roles": {
"creator": true,
"owner": true,
"signed_in": true
}
}
]
}
}
]
[edit2]
I did a lot of multiple tries but still is not printing anything at all!
is like there is no data but there is data... can someone please check my code and see what im I doing wrong? this last one I did step by step with a tutorial, still is not even showing any data, I wish it did so at least I know what to change..
https://gitlab.com/rek2/gohubsbot/-/blob/master/websocketsListen.go
thanks

ok after a lot of debugging, reading, reading, hitting my head and stripping json to make it simples to post on json to go tools... I finally got it..
the main issue?
//the type needs to be a map itself
type PresenceState map[string]State
Also:
// even do I did try this above I never called it the right way before so was giving up and moving on... but I have to use a pointer
for k, o := range *v {}
is working now. I am getting the keys and objets when there is a presence state at login or change so I can map id to usernames

Related

DataTables pagination issues: working with Laravel's pagination model and JSON response

I am trying to build a datatable for a webpage that will eventually hold hundreds of thousands of entries. Because pagination is a must-have, I was given certain local API-calls that implemented Laravel's paginate method. As a result, the server's JSON response has the following format:
{
"current_page": 1,
"data": [{...,
}],
"first_page_url": "http:\/\/127.0.0.1:8000\/api\/test_history\/fullObj?page=1",
"from": 1,
"last_page": 134,
"last_page_url": "http:\/\/127.0.0.1:8000\/api\/test_history\/fullObj?page=134",
"links": [{...,
}],
"next_page_url": "http:\/\/127.0.0.1:8000\/api\/test_history\/fullObj?page=2",
"path": "http:\/\/127.0.0.1:8000\/api\/test_history\/fullObj",
"per_page": 15,
"prev_page_url": null,
"to": 15,
"total": 2000
}
QUESTION
How could DataTables' paging feature implement Laravel's or any pagination model?
What changes in the table initialization, or in general, can be made in order for the page buttons to properly call the appropriate batch of entries?
There should be a way to utilize the above model's next_page_url, first_page_url, etc. properties.
Due to a predicted vast amount of entries, calling the entire object is out of the question, so, unless I am mistaken, DataTables' default paging may not do, because it is not a matter of what is rendered in the front end.
ISSUES
Even though the above JSON response contains all the necessary information for a table pagination, it seems to be conflicting with whatever data the datatable is expecting. So, the following issues surfaced within the webpage:
The total number of pages within the pagination section was not showing
Clicking on any page number, or the next/previous buttons within the pagination section loaded the same entries
The searching and sorting features have stopped filtering the table results
After consulting with documentation, I found that the server's JSON response did not follow DataTables standards, and I had to manipulate some of the data. Here follows the jQuery used to initialize the datatable:
$(document).ready(function() {
$('#test_table').DataTable( {
"order": [[ 8, "desc" ]],
"scrollX": true,
"paging": true,
"lengthMenu": [[ 5, 15, 25, 100, -1 ], [ 5, 15, 25, 100, "All" ]],
"pageLength": 15,
"processing": true,
"serverSide": true,
"ajax": function(data, callback, settings) {
$.get("http://127.0.0.1:8000/api/test_history/fullObj?page=1", {
limit: data.length,
offset: data.start
},
function(json) {
callback({
recordsTotal: json.total,
recordsFiltered: json.total,
data: json.data
});
});
},
"columns": [
{ "data": "id" },
{ "data": "uid" },
{ "data": "dev_type.type" },
{ "data": "registers.id" },
{ "data": "measurements.id" },
{ "data": "created_at" },
{ "data": "updated_at" }
]
});
});
Alternatively:
$(document).ready(function() {
$('#test_table').DataTable( {
"order": [[ 8, "desc" ]],
"scrollX": true,
"paging": true,
"lengthMenu": [[ 5, 15, 25, 100, -1 ], [ 5, 15, 25, 100, "All" ]],
"pageLength": 15,
"processing": true,
"serverSide": true,
"ajax": {
"url": "http://127.0.0.1:8000/api/test_history/fullObj?page=1",
"dataSrc": function(json) {
json.recordsTotal = json.total;
json.recordsFiltered = json.total;
return json.data;
}
},
"columns": [
{ "data": "id" },
{ "data": "uid" },
{ "data": "dev_type.type" },
{ "data": "registers.id" },
{ "data": "measurements.id" },
{ "data": "created_at" },
{ "data": "updated_at" }
]
});
});
These response manipulations fixed the issues of the number of pages not showing, and the incorrect number of page buttons. The rest, however, do persist.
METHODS
Apart from scouring the DataTables' documentation and their website's forum, and combing through the internet for possible solutions, I have tried the following:
Attempting to access an HTML class or ID to create a click event, either by adding the attribute or by creating a handler for the event, possibly not getting it right
Iterating across all numbered buttons, through their DataTables-specific attributes, to dynamically load new entries on click
I have yet to stumble upon a groundbreaking answer, or think of one. Despite these shortcomings, I do believe that the answer is usually simpler than any complex thinking, and in my case it could be related to the data the table is expecting from the server; the JSON's format, or its properties.
Any suggestion would be greatly appreciated.

update one json file with a second json based on third json with jq

I have three JSON files:
vault.json:
{
"aws":
{
"access_key_id": "My-Key-id",
"secret_access_key": "My-Access-Key"
},
"ssl":
{
"crt": "My-Crt",
"key": "My-Key",
"req": "My-Req"
}
}
input.json:
{
".cloud_cpi_key": {
"type": "wildcard_domain",
"configurable": true,
"credential": false,
"value": "vault-supplied-value",
"optional": false
},
".cloud_cpi_secret": {
"type": "wildcard_domain",
"configurable": true,
"credential": false,
"value": "vault-supplied-value",
"optional": false
},
".properties.networking_point_of_entry": {
"type": "selector",
"configurable": true,
"credential": false,
"value": "external_ssl",
"optional": false
},
".properties.networking_point_of_entry.external_ssl.ssl_rsa_certificate": {
"type": "rsa_cert_credentials",
"configurable": true,
"credential": true,
"value": {
"private_key_pem": "vault-supplied-value",
"cert_pem": "vault-supplied-value"
},
"optional": false
}
}
keyfile.json
{
".cloud_cpi_key.value": "aws.access_key_id",
".cloud_cpi_secret": "secret_access_key",
".properties.networking_point_of_entry.external_ssl.ssl_rsa_certificate.value.private_key_pem": "ssl.key",
".properties.networking_point_of_entry.external_ssl.ssl_rsa_certificate.value.cert_pem": "ssl.crt"
}
I'd like to update the second json file, with the values from the first json, based on the third json.
can this be done via JQ to provide output.json?
output.json:
{
".cloud_cpi_key": {
"type": "string",
"configurable": true,
"credential": true,
"value": "My-Key-id",
"optional": false
},
".cloud_cpi_secret": {
"type": "string",
"configurable": true,
"credential": true,
"value": "My-Access-Key",
"optional": false
},
".properties.networking_point_of_entry": {
"type": "selector",
"configurable": true,
"credential": false,
"value": "external_ssl",
"optional": false
},
".properties.networking_point_of_entry.external_ssl.ssl_rsa_certificate": {
"type": "rsa_cert_credentials",
"configurable": true,
"credential": true,
"value": {
"private_key_pem": "My-Key",
"cert_pem": "My-Crt"
},
"optional": false
}
}
I can modify keyfile.json any way I like to make things easier, like
{
"fromkey": "aws.access_key_id"
"tokey": ".cloud_cpi_key.value"
},
{ "fromkey": ....
}
But no values may be placed in keyfile.json, only key names.
And I can modify vault.json, to put things in arrays, or what-have-you, but I cannot change the lowest levels, i.e. I cannot change:
{
"access_key_id": "My-Key-id",
"secret_access_key": "My-Access-Key"
}
I cannot modify input.json.
How can I accomplish this with JQ?
You don't give details about the transformation you have in mind, but if you can specify the algorithm, then rest assured, it can be done in jq.
Perhaps the thing you're missing is how to get jq to read the three files. One way to proceed would be to use the invocation:
jq --argfile keyfile keyfile.json --argfile vault vault.json -f vault.jq input.json
where vault.jq is the file containing your jq program, in which you would refer to the contents of keyfile.json as $keyfile, and similarly for the contents of vault.json
Simplifying keyfile.json
Since you indicate you have some flexibility about the format of keyfile.json, and since it appears to hold path information, I would recommend considering adopting path specifications that can be used directly with the jq builtins getpath and setpath.
For example, consider this format for keyfile.json:
[ [<path in input.json>], [<path in vault.json> ], ... ]
The first two entries corresponding to your example would thus be:
[
[ [".cloud_cpi_key","value"], ["aws","access_key_id"]],
[ [".cloud_cpi_secret"], ["aws", "secret_access_key"]]
]
To use the "vault" as the basis for updates, your jq program would then be the one-liner:
reduce $keyfile[] as $p (.; setpath(($p|.[0]); $vault|getpath($p|.[1])))

How to add tag inside a nest depending on a variable while keeping rest of json

I'm failry new to jq and I've learned to do most things on my own but I'm hitting my head on my keyboard for this one. Look at the following json
JSON:
{
"importType": "Upsert",
"immediateDeployment": false,
"isIgnoreNulls": false,
"isOverwriteNullsOnly": false,
"isPreferredandSync": false,
"outputLayout": {
"fields": [
{
"name": "TEMP_KEY",
"type": "String",
"length": "2000",
"displayName": "TEMP_KEY"
},
{
"name": "LoadSeqNum",
"type": "String",
"length": "2000",
"displayName": "LoadSeqNum"
}
]
}
}
What I'm trying to do is inside of .outputLayout.fields[] I want to create a new pair called "isIdentifier" where it is true if the .outputLayout.fields[].name is LoadSeqNum and false if it's not but I need to keep the rest of the json just as it is. So target should look as following:
Goal:
{
"importType": "Upsert",
"immediateDeployment": false,
"isIgnoreNulls": false,
"isOverwriteNullsOnly": false,
"isPreferredandSync": false,
"outputLayout": {
"fields": [
{
"name": "TEMP_KEY",
"type": "String",
"length": "2000",
"displayName": "TEMP_KEY"
"isIdentifier": false
},
{
"name": "LoadSeqNum",
"type": "String",
"length": "2000",
"displayName": "LoadSeqNum"
"isIdentifier": true
}
]
}
}
I tried this:
jq '.outputLayout.fields[] | . + {"isIdentifier": (if (.name)=="LoadSeqNum" then true else false end)}'
But of course I'm missing all the higher level things. When I try to do:
.outputLayout.fields[].isIdentifier=(if (.outputLayout.fields[].name)=="LoadSeqNum" then true else false end)
I get the whole thing twice, once with both true and the other one with both false. I understand why it's doing that but I'm having a tough time figuring out what would work. Any help or point in right direction?
.outputLayout.fields[] |= (.isIdentifier = (.displayName == "LoadSeqNum") )
Or equivalently but perhaps a little less cryptically:
.outputLayout.fields |= map( .isIdentifier = (.displayName == "LoadSeqNum") )

deserializing JSon with different Parent node names

I'm trying to deserializing a json response from Gitblit using c# and i'm getting it back in the following format
{
"https://localhost/git/libraries/xmlapache.git": {
"name": "libraries/xmlapache.git",
"description": "apache xmlrpc client and server",
"owner": "admin",
"lastChange": "2010-01-28T22:12:06Z",
"hasCommits": true,
"showRemoteBranches": false,
"useTickets": false,
"useDocs": false,
"accessRestriction": "VIEW",
"isFrozen": false,
"showReadme": false,
"federationStrategy": "FEDERATE_THIS",
"federationSets": [
"libraries"
],
"isFederated": false,
"skipSizeCalculation": false,
"skipSummaryMetrics": false,
"size": "102 KB"
},
"https://localhost/git/libraries/smack.git": {
"name": "libraries/smack.git",
"description": "smack xmpp client",
"owner": "admin",
"lastChange": "2009-01-28T18:38:14Z",
"hasCommits": true,
"showRemoteBranches": false,
"useTickets": false,
"useDocs": false,
"accessRestriction": "VIEW",
"isFrozen": false,
"showReadme": false,
"federationStrategy": "FEDERATE_THIS",
"federationSets": [],
"isFederated": false,
"skipSizeCalculation": false,
"skipSummaryMetrics": false,
"size": "4.8 MB"
}
}
and i'm having trouble creating the DataContract due to the parent nodes being named from the pull url and could be anything rather than a standardised name. i only need to deserialize "name":,"description":, "owner":, "lastChange" from underneath them.
so is there a way i can deserialize each dataset while ignoring the parent node all together?

jqgrid datetime column sorting in 24 hour format

I am using a json object as follows from spring MVC to populate the jqgrid by using a json reader.
{
"rows": [
{
"id": 1,
"startTime": "Mar 4 08:00",
"projectId": "Proj_001",
"customerId": "Cust1",
"divisionId": "Div1"
},
{
"id": 2,
"startTime": "Mar 5 15:00",
"projectId": "Proj_001",
"customerId": "Cust1",
"divisionId": "Div1"
},
{
"id": 3,
"startTime": "Apr 14 22:00",
"projectId": "Proj_001",
"customerId": "Cust1",
"divisionId": "Div1"
}
]
}
I need to sort the
startTime
column which comes in 24hour time format (as above in the json object). i tried do some workaround but still unable to succeed.
jqgrid code -
{name:'startTime', index:'startTime', width:90,align:"left",sorttype:'datetime',datefmt:'M d, Y at h:i',sortable: true,editrules:{date:true} },
i will not using AM/PM notation here. i saw in jqgrid.base.js it uses AM/PM notation to sort. can anyone please shed some light on this? i was unable to find any jqgrid example for using date with 24hour time format that will do sort. Thank you in advance.
my jqgrid code -
jq(function(){
jq("#testgrid").jqGrid({
url:"/getall.html",
datatype: 'json',
mtype: 'GET',
colNames:['Id','Start Time','Cust.Id','Div.Id','Proj.Id'],
colModel :[
{name:'id', index:'id', width:90,align:"left",hidden: true},
{name:'startTime', index:'startTime', width:90, align:"left",
sorttype:'datetime', datefmt:'M d h:i',sortable: true,
editrules:{date:true}},
{name:'divisionId', index:'divisionId', width:100,align:"left"},
{name:'departmentId', index:'departmentId', width:120,align:"left"},
{name:'projectId', index:'projectId', width:100,align:"left"},
],
pager: '#testpager',
loadonce: true,
sortname: 'projectId',
sortorder: 'asc',
viewrecords: true,
multiselect: true,
//to hide pager buttons
pgbuttons:false,
recordtext:'',
pgtext:'',
gridview: true,
caption: 'my JQGRID',
jsonReader : {
root: "rows",
page: "page",
total: "total",
records: "records",
repeatitems: false,
cell: "cell",
id: "id"
}
});
//toolbar search
//jq("#testgrid").jqGrid('filterToolbar',
// {stringResult:true,searchOnEnter:false});
});
As I try to reproduce your problem all seems working correct. See the demo live here. If you will sort by the "Start Time" column all work without any problem which I could see.