I wanted to know if it's possible to query the sync gateway buckets using N1QL? Does it behave as a normal couchbase bucket or because of the metadata that sync gateway adds, is it possible to query it only through Rest APIs?
Currently I have a webhooks handler, which keeps a replica of the documents residing under sync gateway buckets. I need to do some aggrgations which need to be pushed back to clients. So, can I do all this heavy lifting directly trhough n1ql on sync gateway or using webhooks which does the aggregations and simply pushes the updated docs to sync gateway is the right option?
PS: The webhooks+Rest APIS option works perfectly for me currently. Just wanted to understand if this hop is necessary or not?
Yes, it is possible to query the sync gateway using N1QL - you just can't change it (update/delete/insert), as it would break the revisions' metadata.
You need to ignore the documents with IDs starting with _sync: and the _sync property of each document, which contains internal metadata. The remaining attributes are your usual document.
Example:
select db.* from db where meta().id not like '_sync:%'
Result:
[
{
"_sync": {
"history": {
"channels": [
null,
null
],
"parents": [
-1,
0
],
"revs": [
"1-b7a15ec4afbb8c4d95e2e897d0ec0a2e",
"2-919b17d3f418100df7298a12ef2a84bb"
]
},
"recent_sequences": [
6,
7
],
"rev": "2-919b17d3f418100df7298a12ef2a84bb",
"sequence": 7,
"time_saved": "2016-05-04T18:54:26.952202911Z"
},
"name": "Document with two revisions"
}
]
Ignoring the _sync attribute:
select name from db where meta().id not like '_sync:%'
Result:
[
{
"name": "Document with two revisions"
}
]
In Couchbase 4.5 (BETA as of today) we can use the object_remove function - although I'd avoid it in favor of the previous more explicit syntax.
select object_remove(db, '_sync') from db where meta().id not like '_sync:%'
Result:
[
{
"$1": {
"name": "Document with two revisions"
}
}
]
I don't know what's your setup currently, but AFAIK, it's perfectly fine to keep querying the bucket throught N1QL while using the REST API for the data changes.
Related
I am syncing a data set with about 300k rows to Phonograph2 and need to make those records available via REST (End Point: /phonograph2/api/search/tables).
My requests looks as following (retrieving records after a certain timestamp):
{
"tableRids": [
"ri.phonograph2.main.table.xxxxxxxxxxxxxxx"
],
"filter": {
"type" : "range",
"range": {
"field": "reco_timestamp",
"gte": "1634408219000"
}
}
}
}
The response ends with:
"nextPageToken": "xxxxxxx"
This leads me to the following questions:
How do I use the "nextPageToken" to retrieve the next set of results?
Can the consumer get a list/array of pages to consume?
Can the number of hits which are displayed until the nextPageToken is written be configured?
We discussed this with our Palantir project support and will use Objects Gateway - as suggested. Thanks for pointing us into this direction.
I followed the instruction in the link below to extract Room objects from Revit models:
https://forge.autodesk.com/blog/new-rvt-svf-model-derivative-parameter-generates-additional-content-including-rooms-and-spaces
I made the changes as instructed and tested the sample Revit file (rac_basic_sample_project.rvt). But, still I don't see the rooms or the viewables (phases). Below is fhe request I post. Am I missing anything?
{
"input": {
"urn": "dXJuOmFkc2sub2JqZWN0czpvcy5vYmplY3Q6YzQ4ZDUxNDNhMDRiNDAxNmI3ODYxY2NlMzQ2ZDkyNjdfZmFjaWxpdHlfOTUvZWIyYzMzNDgtNDAxYS00ZjQ3LTgwM2EtMjM1OGYwYmI0YjY2LnJ2dA"
},
"output": {
"destination": {
"region": "us"
},
"formats": [
{
"type": "svf",
"views": [
"3d"
],
"advanced": {
"generateMasterViews": true
}
}
]
}
}
I just tested the feature and I can see the room data:
The JSON payload seems ok, so try checking the following things:
Make sure you use the x-ads-force header (explained in the blog post you linked to); if you had already processed your Revit model before, triggering a new Model Derivative job would not do anything unless you force the translation
Try using another design (and from a newer version of Revit if possible); in my screenshot I'm using one of the official samples for Revit 2020, although I remember being able to get the room data from older samples as well
The room data is only available in certain "viewables" so make sure you're looking at the right one; for my sample project, for example, the room data is not available in the "{3D}" viewable but it is available in the "Working Drawings" viewable
Azure Function with a complex (List of objects) configuration type is working locally (with that complex type in local.settings.json) but fails to read / create list of objects in Azure (with that complex type in Azure Function configuration settings). I'm looking for the recommended / optimal way to support that across both platforms / methods of access.
This works great in my local.settings.json where I use the configuration builder and pull data out like
var myList = config.GetSection("ConfigurationList").Get<List<MyType>>();
however this doesn't seem to work in Azure Functions?? Now I think that is because in local.settings.json it is a json file and looks like
"ConfigurationList" : [ { "Name": "A", "Value": 2 }, { "Name": "B", "Value": 3 }]
while in Azure Functions it is a setting "ConfigurationList" with the value
[ { "Name": "A", "Value": 2 }, { "Name": "B", "Value": 3 }]
(so there isn't really a "section" in Azure Functions?)
It seems like the "easy" solution to this is to just change the .json to be a quoted string and deserialize the string (and then it would work the same in both places); but that doesn't seem like it would be the "best" (or "recommended" solution)
i.e. something like
"ConfigurationList" : "[ { \"Name\": \"A\", \"Value\": 2 }, { \"Name\": \"B\", \"Value\": 3 }]"
var myList = (List<MyType>)JsonConvert.DeserializeObject(config["ConfigurationList"], typeof(List<MyType>));
Which isn't the worst; but makes the json a bit "not as nice" and doesn't "flow" across the two platforms ... if it is what I have to do, fine; but hoping for a more standard approach / recommendation
As I metioned in the comment, on local you can process local.settings.json as a json file, but when on azure, the value in configuration settings is environment variable. There is no section, it just string.
Please notice that only string values are allowed, and that anything nested will break. Learn how to use nest settings on azure web app(azure functon is based on azure app service sandbox, so it is the same.):
https://learn.microsoft.com/en-us/archive/blogs/waws/asp-net-core-settings-for-azure-app-service
For example, if this is the json structure:
{
"Parent": {
"ChildOne": "C1 from secrets.json",
"ChildTwo": "C2 from secrets.json"
}
}
Then in web app, you should save it like this:
(source: windows.net)
Not sure if you are looking something like this , it seems a list but if it is a simple JObject like
"ConfigurationList" : {
"Name": "A",
"Value": 2
}
Then you can declare ConfigurationList:Name , ConfigurationList:Value in the configuration settings of function app
After creating a Group using POST /v1.0/groups with the body:
{
"description": "hello",
"displayName": "group_for_restore",
"groupTypes": [
"Unified"
],
"mailEnabled": true,
"mailNickname": "group_for_restore",
"securityEnabled": false,
"visibility": "Public"
}
A request to GET /v1.0/groups/{group-id}/planner/plans does not retrieve any plans.
As far as I know, after creating a Group, a Plan will be created too. On the web interface you can see that this plan is correctly created and shown, but it does not appear in JSON response:
{
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#Collection(microsoft.graph.plannerPlan)",
"#odata.count": 0,
"value": []
}
After clicking on it in the web interface you can easily get the plan using the request above.
Is it ok to do such magic steps to fetch plan id?
The Web App is provisioning the Group's Plan the first time you attempt to access it. When creating a Group through the API however, you'll need to create a new Plan yourself:
POST https://graph.microsoft.com/v1.0/planner/plans
Content-type: application/json
{
"owner": "group-id",
"title": "title-value"
}
Keep in mind this important note from the documentation:
When creating a new plan, make a group its owner by simply setting the owner property on a plan object.
Neo4j Version: 3.2.2
Operating System: Ubuntu 16.04
I use getDegree() function in mapping.json file, but the return would always be null; I'm using the dataset neo4j tutorial Movie/Actor dataset.
Output from elasticsearch request
mapping.json
{
"defaults": {
"key_property": "uuid",
"nodes_index": "default-index-node",
"relationships_index": "default-index-relationship",
"include_remaining_properties": true
},
"node_mappings": [
{
"condition": "hasLabel('Person')",
"type": "getLabels()",
"properties": {
"getDegree": "getDegree()",
"getDegree(type)": "getDegree('ACTED_IN')",
"getDegree(direction)": "getGegree('OUTGOING')",
"getDegree('type', 'direction')": "getDegree('ACTED_IN', 'OUTGOING')",
"getDegree-degree": "degree"
}
}
],
"relationship_mappings": [
{
"condition": "allRelationships()",
"type": "type",
}
]
}
Also if I use isOutgoing(), isIncoming(), otherNode function in relationship_mappings properties part, elasticsearch would never load the relationship data from neo4j. I think I probably have some misunderstanding of this sentence only when one of the participating nodes "looking" at the relationship is provided on this page https://github.com/graphaware/neo4j-framework/tree/master/common#inclusion-policies
mapping.json
{
"defaults": {
"key_property": "uuid",
"nodes_index": "default-index-node",
"relationships_index": "default-index-relationship",
"include_remaining_properties": true
},
"node_mappings": [
{
"condition": "allNodes()",
"type": "getLabels()"
}
],
"relationship_mappings": [
{
"condition": "allRelationships()",
"type": "type",
"properties": {
"isOutgoing": "isOutgoing()",
"isIncomming": "isIncomming()",
"otherNode": "otherNode"
}
}
]
}
BTW, is there any page that list all of the functions that we can use in mapping.json? I know two of them
github.com/graphaware/neo4j-framework/tree/master/common#inclusion-policies
github.com/graphaware/neo4j-to-elasticsearch/blob/master/docs/json-mapper.md
but it seems there are more, since I can use getType(), which hasn't been listed in any of the above pages.
Please let me know if I can provide any further help to solve the problem
Thanks!
The getDegree() function is not available to use, in contrary to getType(). I will explain why :
When the mapper (the part responsible to create a node or relationship representation as ES document ) is doing its job, it receive a DetachedGraphObject being a detached node or relationship.
The meaning of detached is that it is happening outside of a transaction and thus query operations are not available against the database anymore. The getType() is available because it is part of the relationship metadata and it is cheap, however if we would want to do the same for getDegree() this can be seriously more costly during the DetachedObject creation (which happen in a tx) depending on the number of different types etc.
This is however something we are working on, by externalising the mapper in a standalone java application coupled with a broker like kafa, rabbit,.. between neo and this app. We would not, however offer the possibilty to requery the graph in the current version of the module as it can have serious performance impacts if the user is not very careful.
As last, the only suggestion I can give you is to keep a property on your node with the updates of degrees you need to replicate to ES.
UPDATE
Regarding this part of the documentation :
For Relationships only when one of the participating nodes "looking" at the relationship is provided:
This is used only when not using the json definition, so you can use one or the other. the json definition has been added later as addition and both cannot be used together.
For answering this part, it means that the nodes of the incoming or outgoing side, depending on the definition, should be included in the inclusion policy for nodes, like hasLabel('Employee') || hasProperty('form') || getProperty('age', 0) > 20 . If you have an allNodes policy then it is fine.