I'm trying to retrieve data from a WEB API rest server.
https://stat-xplore.dwp.gov.uk/webapi/online-help/Open-Data-API.html
Which stores the data in Cubes.
I understand the basics and am successfully retrieving data. However I want to use a Postcode, (ZIP Code) grouping dimension.
This dimension groups postcodes by their first 3 characters. CF1, CF2 etc.
The schema has this as the definition for sending the request for this dimension.
["str:group:UC_Households:X_Geography+%28postcode%29"]
I'm assuming I'm meant to replace the end part with the Postcode group that I'm interested in and enclose the postcode group with some special characters.
So far I have tried
["str:group:UC_Households:X_Geography:CF1"]
["str:group:UC_Households:X_Geography:'CF1'"]
["str:group:UC_Households:X_Geography:#CF1#"]
["str:group:UC_Households:X_Geography:CF1%"]
["str:group:UC_Households:X_Geography#CF1#"]
["str:group:UC_Households:X_GeographyCF1"]
["str:group:UC_Households:X_Geography:CF1"]
the complete JSON is
{ "database" : "str:database:UC_Households",
"measures" : [ "str:count:UC_Households:V_F_UC_HOUSEHOLDS" ],
"dimensions" : [
[ "str:field:UC_Households:V_F_UC_HOUSEHOLDS:HNFAMILY_TYPE" ],
["str:group:UC_Households:X_Geography:CF1"]
]
}
Any help would be appreciated.
How to easily generate the query JSON for the Stat-Xplore API
Log into https://stat-xplore.dwp.gov.uk
Open the data set and table you want
In the top-right hand corner, find the "Download Table" tool
In the drop-down box, select "Open Data API Query (.json) and click "Go"
This will download a JSON file with the parameters for the query that describes the table you're looking at in your web browser. An example is shown below:
{
"database" : "str:database:UC_Monthly",
"measures" : [ "str:count:UC_Monthly:V_F_UC_CASELOAD_FULL" ],
"recodes" : {
"str:field:UC_Monthly:F_UC_DATE:DATE_NAME" : {
"map" : [ [ "str:value:UC_Monthly:F_UC_DATE:DATE_NAME:C_UC_DATE:202010" ] ],
"total" : false
},
"str:field:UC_Monthly:V_F_UC_CASELOAD_FULL:EMPLOYMENT_CODE" : {
"map" : [ [ "str:value:UC_Monthly:V_F_UC_CASELOAD_FULL:EMPLOYMENT_CODE:C_UC_EMPLOYMENT:0" ], [ "str:value:UC_Monthly:V_F_UC_CASELOAD_FULL:EMPLOYMENT_CODE:C_UC_EMPLOYMENT:1" ] ],
"total" : true
},
"str:field:UC_Monthly:V_F_UC_CASELOAD_FULL:PARLC_CODE" : {
"map" : [ [ "str:value:UC_Monthly:V_F_UC_CASELOAD_FULL:PARLC_CODE:V_C_MASTERGEOG11_PARLC_TO_REGION:E14000541" ], [ "str:value:UC_Monthly:V_F_UC_CASELOAD_FULL:PARLC_CODE:V_C_MASTERGEOG11_PARLC_TO_REGION:E14000542" ], [ "str:value:UC_Monthly:V_F_UC_CASELOAD_FULL:PARLC_CODE:V_C_MASTERGEOG11_PARLC_TO_REGION:W07000041" ] ],
"total" : true
}
},
"dimensions" : [ [ "str:field:UC_Monthly:V_F_UC_CASELOAD_FULL:PARLC_CODE" ], [ "str:field:UC_Monthly:F_UC_DATE:DATE_NAME" ], [ "str:field:UC_Monthly:V_F_UC_CASELOAD_FULL:EMPLOYMENT_CODE" ] ]
}
The API table endpoint docs give more info.
An explanation for OP's issue
I'm also attempting to use this API. (I happen to be using Python requests library.)
If I make a request to the /v1/table endpoint where one dimension is of type GROUP (rather than FIELD)
measures=["str:count:UC_Monthly:V_F_UC_CASELOAD_FULL"],
dimensions=[["str:field:UC_Monthly:F_UC_DATE:DATE_NAME"],
["str:group:UC_Monthly:X_Geography+%28residence-based%29"]
]
then this error 422 is returned in the HTTP response:
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://stat-xplore.dwp.gov.uk/webapi/rest/v1/table
This is the payload of the response, which describes the error in more detail:
{
"message":"Unexpected schema component type 'group' for 'str:group:UC_Monthly:X_Geography+%28residence-based%29'. Expected one of type [field, valueset, udf].",
"errorType":"UNEXPECTED_SCHEMA_COMPONENT_TYPE_EXCEPTION",
"component" : "str:group:UC_Monthly:X_Geography+%28residence-based%29",
"type" : "group",
"expectedTypes" : [ "field", "valueset", "udf" ]
}
If I understand this correctly, it's not complaining about the formatting of the string, it's saying that only FIELD, VALUESET and UDF objects are expected.
If you look at the schema endpoint one can retrieve info about the possible groups, fields, etc.
{
'id': 'str:group:UC_Monthly:X_Geography+%28postcode%29',
'label': 'Geography (postcode)',
'location': 'https://stat-xplore.dwp.gov.uk/webapi/rest/v1/schema/str:group:UC_Monthly:X_Geography+%2528postcode%2529',
'type': 'GROUP'
}
Related
Below is my query result. i want the json out put as :
[
{id:1,
name:a,
category : [
{name:balakrishnan},
{name:nikhil}
]
},
{id:2,
name:b,
category : [
{name:midhun},
{name:amith}
]
}
]
I have done this with async parallel, i just want to know where we can do in another approach like map functions
Trying out the Google Maps Distance Matrix API for calculating the time between the origin and destination and I'm using this API
https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=32.988777,xxxx&destinations=32.988777,xxxx&key=myAPIKey
(xxxx - I'm providing a valid value just masking it on here)
I am getting a response as
{
"destination_addresses" : [ "Plano, TX 75024, USA" ],
"origin_addresses" : [ "Coppell, TX 75019, USA" ],
"rows" : [
{
"elements" : [
{
"distance" : {
"text" : "13.4 mi",
"value" : 21636
},
"duration" : {
"text" : "14 mins",
"value" : 833
},
"status" : "OK"
}
]
}
],
"status" : "OK"
}
But I'm unable to correlate the req and response as the response is missing the long/lat, How do I solve this issue?
Per official documentation: DistanceMatrix service does not include lat/lng element in response. See all available elements here: https://developers.google.com/maps/documentation/distance-matrix/intro#DistanceMatrixResponses
Nonetheless, looking at your request, it appears that you already supplied lat/lng in your input. On the other hand if you supplied address as origins/destinations and aiming to extract the corresponding coordinates, then you can simply do forward Geocode afterwards.
I have to insert documents in my Mongo database by using Ruby (not on Rails, on Notepad++), many documents have duplicates with some modifications.
I want to write a script which use a json file, read it, import it in MongoDB by checking if each documents do not have any duplicate in the database, and if there is a duplicate I want to combine it if it contains any additional information:
Such as :
Document 1
{ "Name" : "Lila",
"Files":
[
{ "Name": "File1", "Date" : "05-11-2017"},
{ "Name": "File2", "Date" : "26-03-2018"}
]
}
Document 2
{ "Name" : "Lila",
"Files":
[
{ "Name": "File3", "Date" : "26-03-2018"}
]
}
Combine them to have:
{ "Name" : "Lila",
"Files":
[
{ "Name": "File1", "Date" : "05-11-2017"},
{ "Name": "File2", "Date" : "26-03-2018"},
{ "Name": "File3", "Date" : "26-03-2018"}
]
}
I founded that it is possible in mongo shell thanks to the aggregation-accumulation $mergeObjects, but in Ruby it do not seems to exists.
You can use all the operators in ruby, too. You need to get the underlying collection object first.
require 'mongo'
db = Mongo::Connection.new.db("mydb")
coll = db.collection('posts')
coll.aggregate([
{"$project" => {"last_name" => 1, "first_name" => 1 }},
{"$match" => {"last_name" => "Jones"}}
])
This is an example pipeline. You can give the same aggregation pipeline that worked for you on the mongo shell to aggregate.
For more information, refer to the MongoDB Ruby driver documentation:
http://www.rubydoc.info/gems/mongo/1.8.2/Mongo%2FCollection%3Aaggregate
I am new to Json stuff i.e. JSON PATCH.
I have scenario where I need to figure out between two version of Json files of same object, for that I am using json-patch-master.
But unfortunately the patch generated interpreting it differently i.e. the order differently hence getting unexpected/invalid results.
Could anyone help me how to preserve the order while generating Json Patch ?
**Here is the actual example.
Original Json file :**
[ {
"name" : "name1",
"roolNo" : "1"
}, {
"name" : "name2",
"roolNo" : "2"
}, {
"name" : "name3",
"roolNo" : "3"
}, {
"name" : "name4",
"roolNo" : "4"
} ]
**Modified/New Json file: i.e. removed 2nd node of original file.**
[ {
"name" : "name1",
"roolNo" : "1"
}, {
"name" : "name3",
"roolNo" : "3"
}, {
"name" : "name4",
"roolNo" : "4"
} ]
**Patch/Diff Generated :**
[ {"op":"remove","path":"/3"},
{"op":"replace","path":"/1/name","value":"name3"},
{"op":"replace","path":"/1/roolNo","value":"3"},
{"op":"replace","path":"/2/name","value":"name4"},
{"op":"replace","path":"/2/roolNo","value":"4"}]
Very time I generate Diff/Patch it is giving different path/diff results.
And moreover the interpretation is different i.e. order is not preserving.
**Is there any way to get expected results i.e. [ {"op":"remove","path":"/1"} ] , in other words generated a patch/diff based some order so will get what is expected. ?
How to handle this kind of scenario ?**
Please help me.
Thank you so much.
~Shyam
We are currently working on this issue in Starcounter-Jack/JSON-Patch.
It seems to work nice with native Array.Observe- http://jsfiddle.net/tomalec/p4s7aw96/.
Try Starcounter-Jack/JSON-Patch issues/65_ArrayObserve branch
we will release it as new version once shim and performance will be checked.
Feel free to add you comments at JSON-Patch issue board
I have a basic Json question - I have a JSON file. Every object in this file has columns repeated.
[
{
id: 1,
name: "ABCD"
},
{
id: 2,
name: "ABCDE"
},
{
id: 3,
name: "ABCDEF"
}
]
For optimization I was thinking to remove repeated column names.
{
"cols": [
"id",
"name"
],
"rows": [
[
"1",
"ABCD"
],
[
"2",
"ABCDE"
]
]
}
What I am trying to understand is - is this a better approach? Are there any disadvantages of this format? Say for writing unit tests?
EDIT
The second case (after your editing) is valid json. You can derive it to the following class using json2csharp
public class RootObject
{
public List<string> cols { get; set; }
public List<List<string>> rows { get; set; }
}
The very important point to note about a valid json is that it has no other way but to repeat the column names (or, keys in general) to represent values in json. You can test the validity of your json putting it # jsonlint.com
But if you want to optimize json by compressing it using some compression library like gzip (likewise), then I would recommend Json.HPack.
According to this format, it has many compression levels ranging from 0 to 4 (4 is the best).
At compression level 0:
you have to remove keys (property names) from the structure creating a header on index 0 with each property name. Then your compressed json would look like:
[
[
"id",
"name"
],
[
1,
"ABCD"
],
[
2,
"ABCDE"
],
[
3,
"ABCDEF"
]
]
In this way, you can compress your json at any levels as you want. But in order to work with any json library, you must have to decompress it to valid json first like the one you provided earlier with repeated property names.
For your kind information, you can have a look at the comparison between different compression techniques:
{
"cols": [
"id",
"name"
],
"rows": [
"1",
"ABCD"
], [
"2",
"ABCDE"
], [
"3",
"ABCDEF"
]
}
In this approach it will be hard to determine which value stand for which item (id,name). Your first approach was good if you use this JSON for communication.
A solution for it, is use any type (by your preference) of Object-Relational-Mapper,
By that, you can compress your JSON data and still using legible structure/code.
Please, see this article: What is "compressed JSON"?