How to calculate max in sth-comet? - fiware

sth-comet offers the possibility of calculate max, min and other functions, as described [here]https://github.com/telefonicaid/fiware-sth-comet/blob/master/doc/manuals/aggregated-data-retrieval.md
But I have tried different types and it doesn't gives the aggregated result.
A simplified version of my entity (I use the attribute temperature in this example) is:
{
"id": "Beach:27",
"type": "Beach",
"flag": {
"type": "Property",
"value": "Verde"
},
"temperature": {
"type": "Number",
"value": 45
}
I have make this query that should give the maximum value:
http://{{sth-comet}}/STH/v1/contextEntities/type/Beach/id/Beach:27/attributes/temperature?aggrMethod=max&hLimit=100&hOffset=0
but the result is not the max but all the changes of the attribute:
{
"contextResponses": [
{
"contextElement": {
"attributes": [
{
"name": "temperature",
"values": [
{
"recvTime": "2019-09-15T18:32:18.166Z",
"attrType": "Number",
"attrValue": "43"
},
{
"recvTime": "2019-09-15T18:32:24.645Z",
"attrType": "Number",
"attrValue": "44"
},
{
"recvTime": "2019-09-15T18:32:28.931Z",
"attrType": "Number",
"attrValue": "45"
}
]
}
],
"id": "Beach:27",
"isPattern": false,
"type": "Beach"
},
"statusCode": {
"code": "200",
"reasonPhrase": "OK"
}
}
]
}
Which is the type that the property must have to work correctly? I tried "Number", "integer","string", "Property" but I don't obtain the "max" value.
Thank you for your time

The requests for aggregated time series context information can use the following query parameters:
aggrMethod: (mandatory) You can use max, min, sum and sum2.
aggrPeriod: (mandatory) you can use month, day, hour, minute and second.
dateFrom and dateTo: (optional) to range of data you query.
In your request you don't put the mandatory parameter (aggrPeriod). Can you test something else:
( http://{{sth-comet}}/STH/v1/contextEntities/type/Beach/id/Beach:27/attributes/temperature?aggrMethod=max&aggrPeriod=second&dateFrom=2019-09-15T00:00:00.000Z&dateTo=2019-09-15T23:59:59.999Z )
https://github.com/telefonicaid/fiware-sth-comet/blob/master/doc/manuals/aggregated-data-retrieval.md

Related

Azure data factory appending to Json

Wanted to pick your brains on something
So, in Azure data factory, I am running a set of activities which at the end of the run produce a json segment
{"name":"myName", "email":"email#somewhere.com", .. <more elements> }
This set of activities occurs in a loop - Loop Until activity.
My goal is to have a final JSON object like this:
"profiles":[
{"name":"myName", "email":"email#somewhere.com", .. <more elements> },
{"name":"myName", "email":"email#somewhere.com", .. <more elements> },
{"name":"myName", "email":"email#somewhere.com", .. <more elements> },
...
{"name":"myName", "email":"email#somewhere.com", .. <more elements> }
]
That is a concatenation of all the individual ones.
To put in perspective, each individual item is a paged data from a rest api - and all them constitute the final response. I have no control over how many are there.
I understand how to concatenate individual items using 2 variables
jsonTemp = #concat(finalJson, individualResponse)
finalJson = jsonTemp
But, I do not know how to make it all under the single roof "profiles" afterwards.
So this is a bit of a hacky way of doing it and happy to hear a better solution.
I'm assuming you have stored all your results in an array variable (let's call this A).
First step is to find the number of elements in this array. You can
do this using the length(..) function.
Then you go into a loop, interating a counter variable and
concatenating each of the elements of the array making sure you add
a ',' in between each element. You have to make sure you do not add
the ',' after the last element(You will need to use an IF condition
to check if your counter has reached the length of the array. At the
end of this you should have 1 string variable like this.
{"name":"myName","email":"email#somewhere.com"},{"name":"myName","email":"email#somewhere.com"},{"name":"myName","email":"email#somewhere.com"},{"name":"myName","email":"email#somewhere.com"}
Now all you need to do this this expression when you are pushing the
response anywhere.
#json(concat('{"profiles":[',<your_string_variable_here>,']}'))
I agree with #Anupam Chand and I am following the same process with a different Second step.
You mentioned that your object data comes from API pages and to end the until loop you have to give a condition to about the number of pages(number of iterations).
This is my pipeline:
First I have initilized a counter and used that counter each web page URL and in until condition to meet a certain number of pages. As ADF do not support self referencing variables, I have used another temporary variable to increment the counter.
To Store the objects of each iteration from web activity, I have created an array variable in pipeline.
Inside ForEach, use append variable activity to append each object to the array like below.
For my sample web activity the dynamic content will be #activity('Data from REST').output.data[0]. for you it will be like #activity('Web1').output. change it as per your requirement.
This is my Pipeline JSON:
{
"name": "pipeline3",
"properties": {
"activities": [
{
"name": "Until1",
"type": "Until",
"dependsOn": [
{
"activity": "Counter intialization",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"expression": {
"value": "#equals('3', variables('counter'))",
"type": "Expression"
},
"activities": [
{
"name": "Data from REST",
"type": "WebActivity",
"dependsOn": [
{
"activity": "counter in temp variable",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "https://reqres.in/api/users?page=#{variables('counter')}",
"type": "Expression"
},
"method": "GET"
}
},
{
"name": "counter in temp variable",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "tempCounter",
"value": {
"value": "#variables('counter')",
"type": "Expression"
}
}
},
{
"name": "Counter increment using temp",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Data from REST",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "counter",
"value": {
"value": "#string(add(int(variables('tempCounter')),1))",
"type": "Expression"
}
}
},
{
"name": "Append web output to array",
"type": "AppendVariable",
"dependsOn": [
{
"activity": "Counter increment using temp",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "arr",
"value": {
"value": "#activity('Data from REST').output.data[0]",
"type": "Expression"
}
}
}
],
"timeout": "0.12:00:00"
}
},
{
"name": "Counter intialization",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "counter",
"value": {
"value": "#string('1')",
"type": "Expression"
}
}
},
{
"name": "To show res array",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Until1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "res_show",
"value": {
"value": "#variables('arr')",
"type": "Expression"
}
}
}
],
"variables": {
"arr": {
"type": "Array"
},
"counter": {
"type": "String"
},
"tempCounter": {
"type": "String"
},
"res_show": {
"type": "Array"
},
"arr_string": {
"type": "String"
}
},
"annotations": []
}
}
Result in an array variable:
You can access this array by the variable name. If you want the output to be like yours, you can use the below dynamic content.
#json(concat('{','"profile":',string(variables('res_show')),'}')))
However, if you want to store this in a variable, you have to wrap it in #string() as currently, ADF variables only supports int, string and array type only.

Azure Cost Management API does not allow me to select columns

I tried to use the Azure Cost Management - Query Usage API to get details (certain columns) on all costs for a given subscription. The body I use for the request is
{
"type": "Usage",
"timeframe": " BillingMonthToDate ",
"dataset": {
"granularity": "Daily",
"configuration": {
"columns": [
"MeterCategory",
"CostInBillingCurrency",
"ResourceGroup"
]
}
}
But the response I get back is this:
{
"id": "xxxx",
"name": "xxxx",
"type": "Microsoft.CostManagement/query",
"location": null,
"sku": null,
"eTag": null,
"properties": {
"nextLink": null,
"columns": [
{
"name": "UsageDate",
"type": "Number"
},
{
"name": "Currency",
"type": "String"
} ],
"rows": [
[
20201101,
"EUR"
],
[
20201102,
"EUR"
],
[
20201103,
"EUR"
],
...
]
}
The JSON continues listing all the dates with the currency.
When I use the dataset.aggregation or dataset.grouping clauses in the JSON, I do get costs returned in my JSON but then I don't get the detailed column information that I want. And of course it is not possible to combine these 2 clauses with the dataset.columns clause. Anyone have any idea what I'm doing wrong?
I found a solution without using the dataset.columns clause (which might just be a faulty clause?). By grouping the data according tot the columns I want, I can also get the data for those column values:
{
"type": "Usage",
"timeframe": "BillingMonthToDate",
"dataset": {
"granularity": "Daily",
"aggregation": {
"totalCost": {
"name": "PreTaxCost",
"function": "Sum"
}
},
"grouping": [
{
"type": "Dimension",
"name": "SubscriptionName"
},
{
"type": "Dimension",
"name": "ResourceGroupName"
}
,
{
"type": "Dimension",
"name": "meterSubCategory"
}
,
{
"type": "Dimension",
"name": "MeterCategory"
}
]
}

Check if keyword exists in a JSON response - Azure Logic Apps

In my Azure Logic App, I have an action to get rows from SQL database as follows.
And sample output (body) of this as follows,
{
"#odata.context": "https://logic-apis-southeastasia.azure-apim.net/apim/sql/5bb78f1b756e4b6097a8bccb6be8dae7/$metadata#datasets('virtueagintegrationssqldbsv-dev2.database.windows.net%2CLearnIntegrationDB-dev2')/tables('%5Bdbo%5D.%5BLearnEmployeeExamData%5D')/items",
"value": [
{
"#odata.etag": "",
"ItemInternalId": "ddf29856-4452-4511-a041-83a4bcf3e8fc",
"EXAMSTART": "YES",
"EXAMRESULT": "YES"
},
{
"#odata.etag": "",
"ItemInternalId": "b5a0261b-c5bf-4f14-8a87-a6acd3aaa26b",
"EXAMSTART": "YES",
"EXAMRESULT": "YES"
},
{
"#odata.etag": "",
"ItemInternalId": "7035458b-605d-431e-a352-dc91261f2a59"
},
{
"#odata.etag": "",
"ItemInternalId": "648d4c06-c3e0-45a9-b656-1aab485d12fd"
}
]
}
Is there expression to check at least one item has "EXAMSTART": "YES" from the item list "values" as shown in above response??
Ex: For above response it should output True as it's having two such items.
You can use Data Operations-> Filter Array step to get only those items with EXAMSTART: "YES":
and then use length to evaluate whether there's any array item returned from Filter Array:
code view:
"Condition": {
"actions": {},
"expression": {
"and": [
{
"greater": [
"#length(body('Filter_array'))",
0
]
}
]
},
"runAfter": {
"Filter_array": [
"Succeeded"
]
},
"type": "If"
},
"Filter_array": {
"inputs": {
"from": "#body('Get_rows_(V2)')?['value']",
"where": "#equals(item()?['EXAMSTART'], 'YES')"
},
"runAfter": {
"Get_rows_(V2)": [
"Succeeded"
]
},
"type": "Query"
},

Getting readable results from Wikidata

Ok so I'm trying to get information from Wikidata about movies, take this movie for example: https://www.wikidata.org/wiki/Q24871
On the page the data is clearly displayed in a readable format, however when you trying to extract it via the API you get this: https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q24871
Here is a section from it:
"P272": [
{
"id": "q24871$4721C959-0FCF-49D4-9265-E4FAC217CB6E",
"mainsnak": {
"snaktype": "value",
"property": "P272",
"datatype": "wikibase-item",
"datavalue": {
"value": {
"entity-type": "item",
"numeric-id": 775450
},
"type": "wikibase-entityid"
}
},
"type": "statement",
"rank": "normal"
},
{
"id": "q24871$31777445-1068-4C38-9B4B-96362577C442",
"mainsnak": {
"snaktype": "value",
"property": "P272",
"datatype": "wikibase-item",
"datavalue": {
"value": {
"entity-type": "item",
"numeric-id": 3041294
},
"type": "wikibase-entityid"
}
},
"type": "statement",
"rank": "normal"
},
{
"id": "q24871$08009F7A-8E54-48C3-92D9-75DEF4CF3E8D",
"mainsnak": {
"snaktype": "value",
"property": "P272",
"datatype": "wikibase-item",
"datavalue": {
"value": {
"entity-type": "item",
"numeric-id": 646968
},
"type": "wikibase-entityid"
}
},
"type": "statement",
"rank": "normal"
},
{
"id": "q24871$CA53B5EB-1041-4701-A36E-7C348FAC984E",
"mainsnak": {
"snaktype": "value",
"property": "P272",
"datatype": "wikibase-item",
"datavalue": {
"value": {
"entity-type": "item",
"numeric-id": 434841
},
"type": "wikibase-entityid"
}
},
"type": "statement",
"rank": "normal",
"references": [
{
"hash": "50f57a3dbac4708ce4ae4a827c0afac7fcdb4a5c",
"snaks": {
"P143": [
{
"snaktype": "value",
"property": "P143",
"datatype": "wikibase-item",
"datavalue": {
"value": {
"entity-type": "item",
"numeric-id": 11920
},
"type": "wikibase-entityid"
}
}
]
},
"snaks-order": [
"P143"
]
}
]
}
],
The problem is I'm not sure how to convert sections like that into readable text. I get the API is calling a link between a class and its properties using unique IDs but I'm still stuck.
Is this actually possible at present or am I barking up the wrong tree?
What you should be looking for are the numeric-ids in each statements and add a leading Q to recover your wikidata ids, which should result to ['Q775450', 'Q3041294', 'Q646968', 'Q434841', 'Q11920']
[update: you can now directly access the Q id at mainsnak.datavalue.value.id, instead of having to build it from the numeric-id]
This can be done using wikibase-sdk (a JS lib I developed) wbk.simplify.claims function
Once you got those ids, you just need to request entities labels using the wbgetentities API:
https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q775450|Q3041294|Q646968|Q434841|Q11920&format=json&props=labels
you can even get results for only some languages, using the languages parameter: https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q775450|Q3041294|Q646968|Q434841|Q11920&format=json&props=labels&languages=en|de|fr
I see an accepted answer, but initially interpreted the question differently. Basically asking to have the same output in JSON one sees on the Wikidata item page.
SPARQL query with JSON output for above case:
https://query.wikidata.org/sparql?query=SELECT%20%3FwdLabel%20%3Fps_Label%20%3FwdpqLabel%20%3Fpq_Label%20%7B%0A%20%20VALUES%20(%3Fcompany)%20%7B(wd%3AQ24871)%7D%0A%0A%20%20%3Fcompany%20%3Fp%20%3Fstatement%20.%0A%20%20%3Fstatement%20%3Fps%20%3Fps_%20.%0A%0A%20%20%3Fwd%20wikibase%3Aclaim%20%3Fp.%0A%20%20%3Fwd%20wikibase%3AstatementProperty%20%3Fps.%0A%0A%20%20OPTIONAL%20%7B%0A%20%20%3Fstatement%20%3Fpq%20%3Fpq_%20.%0A%20%20%3Fwdpq%20wikibase%3Aqualifier%20%3Fpq%20.%0A%20%20%7D%0A%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%7D&format=json
I use the Wikidata Query Front End to get my query straight and to check the results. Then use the </> Code button... explaining why you're seeing so much unnecessary whitespace above.
See also:
wikidata get all properties with labels and values of an item
SPARQL query service - Interfacing
Ok so I haven't found a solution to using the This is the "wbgetentities" system I have found that you can use the "parse" command to get the html structure.
https://www.wikidata.org/w/api.php?action=parse&page=Q24871
While it still going to need some processing its much easier than the previous solution.

Facets tokenize tags with spaces. Is there a solution?

I have some problem with facets tokenize tags with spaces.
I have the following mappings:
curl -XPOST "http://localhost:9200/pictures" -d '
{
"mappings" : {
"pictures" : {
"properties" : {
"id": { "type": "string" },
"description": {"type": "string", "index": "not_analyzed"},
"featured": { "type": "boolean" },
"categories": { "type": "string", "index": "not_analyzed" },
"tags": { "type": "string", "index": "not_analyzed", "analyzer": "keyword" },
"created_at": { "type": "double" }
}
}
}
}'
And My Data is:
curl -X POST "http://localhost:9200/pictures/picture" -d '{
"picture": {
"id": "4defe0ecf02a8724b8000047",
"title": "Victoria Secret PhotoShoot",
"description": "From France and Italy",
"featured": true,
"categories": [
"Fashion",
"Girls",
],
"tags": [
"girl",
"photoshoot",
"supermodel",
"Victoria Secret"
],
"created_at": 1405784416.04672
}
}'
And My Query is:
curl -X POST "http://localhost:9200/pictures/_search?pretty=true" -d '
{
"query": {
"text": {
"tags": {
"query": "Victoria Secret"
}
}
},
"facets": {
"tags": {
"terms": {
"field": "tags"
}
}
}
}'
The Output result is:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
},
"facets" : {
"tags" : {
"_type" : "terms",
"missing" : 0,
"total" : 0,
"other" : 0,
"terms" : [ ]
}
}
}
Now, I got total 0 in facets and total: 0 in hits
Any Idea Why its not working?
I know that when I remove the keyword analyzer from tags and make it "not_analyzed" then I get result.
But there is still a problem of case sensitive.
If I run same above query by removing the keyword analyzer then I get the result which is:
facets: {
tags: {
_type: terms
missing: 0
total: 12
other: 0
terms: [
{
term: photoshoot
count: 1
}
{
term: girl
count: 1
}
{
term: Victoria Secret
count: 1
}
{
term: supermodel
count: 1
}
]
}
}
Here Victoria Secret is case sensitive in "not_analyzed" but it takes space in count, but when I query with lowercase as "victoria secret" it doesn't give any results.
Any suggestions??
Thanks,
Suraj
The first examples are not totally clear to me. If you use the KeywordAnalyzer it means that the field will be indexed as it is, but then it makes much more sense to just not analyze the field at all, which is the same. The mapping you posted contains both
"index": "not_analyzed", "analyzer": "keyword"
which doesn't make a lot of sense. If you are not analyzing the field why would select an analyzer for it?
Apart from this, of course if you don't analyze the field the tag Victoria Secret will be indexed as it is, thus the query victoria secret won't match. If you want it to be case-insensitive you need to define a custom analyzer which uses the KeyworkTokenizer, since you don't want to tokenize it and the LowercaseTokenFilter. You can define a custom analyzer through the index settings analysis section and then use it in your mapping. But that way the facet would be always lowercase, which is something that you don't like I guess. That's why it's better to define a multi field and index the field using two different text analysis, one for the facet and one for search.
You can create the index like this:
curl -XPOST "http://localhost:9200/pictures" -d '{
"settings" : {
"analysis" : {
"analyzer" : {
"lowercase_analyzer" : {
"type" : "custom",
"tokenizer" : "keyword",
"filter" : [ "lowercase"]
}
}
}
},
"mappings" : {
"pictures" : {
"properties" : {
"id": { "type": "string" },
"description": {"type": "string", "index": "not_analyzed"},
"featured": { "type": "boolean" },
"categories": { "type": "string", "index": "not_analyzed" },
"tags" : {
"type" : "multi_field",
"fields" : {
"tags": { "type": "string", "analyzer": "lowercase_analyzer" },
"facet": {"type": "string", "index": "not_analyzed"},
}
},
"created_at": { "type": "double" }
}
}
}
}'
Then the custom lowercase_analyzer will be applied by default to the text query too when you search on that field, so that you can either search for Victoria Secret or victoria secret and get the result back. You need to change the facet part and make the facet on the new tags.facet field, which is not analyzed.
Furthermore, you might want to have a look at the match query since the text query has been deprecated with the latest elasticsearch version (0.19.9).
I think this make some sense to my answer
https://gist.github.com/2688072