When i write view in couchbase to return whole doc it also return _sync":{} data, is there any way to remove this data from response.
here is my view function:-
function map(doc, meta) {
if (doc.type == 'user' && doc.user_id) {
emit(doc.user_id, doc);
}
}
first off, you shouldn't ever have to emit the whole doc. This makes the index bigger on disk, and it's redudant since you can get the whole doc easily from a view row (the doc id is always included, and the SDKs will transparently fetch it for you usually).
in your case though, you may actually need that second part of emit. Select the attributes you are interested in and emit them in an array (like emit(doc.user_id, [doc.attributeA, doc.attributeB])), thus effectively "filtering out" _sync.
only problem is that if later on you add an attributeC to your users, it won't automatically be included (so it filters out attributes not explicitly listed in the map function). does that make sense?
Related
I have phonorgraph object with billions of rows and we are querying it through object set service
for example, I want to get all DriverLicences from certain city.
#Function()
public getDriverLicences(city: string): ObjectSet<DriverLicences> {
let drivers = Objects.search().DriverLicences().filter(row => row.city.exactMatch(city));
return drivers ;
}
I am facing this error when I am trying query it from slate:
ERROR 400: {"errorCode":"INVALID_ARGUMENT","errorName":"ObjectSet:PagingAboveConfiguredLimitNotAllowed","errorInstanceId":"0000-000","parameters":{}}
I understand that I am probably retrieving more than 100 000 results but I need all the results because of the implemented logic in the front is a complex slate dashboard built by another team that we cannot re-factor.
The issue here is that, specifically in the Slate <> Function connector, there is a "translation layer" that serializes the contents of the object set and provides a response data structure that materializes the property:value pairs for each object in the set.
This clearly doesn't work for large object sets where throwing so much data into the browser is likely to overwhelm the resources allocated to the tab.
From context it seems like you might be migrating an existing Slate app over to Functions; in the current version, how is the query limiting the number of results returned? It certainly must not be returning several 100 thousand results for further processing on the front end? (And if so, that might be an anti-pattern to consider addressing).
As for options that you could currently explore, you can sort your object set and then specify a smaller limit to return:
Objects.search().DriverLicences().filter(row => row.city.exactMatch(city)).orderBy(date_of_issue).take(100)
You'll find a few more details in the Functions documentation Reference entry on Ontology API: Object Sets in the section on Ordering and limiting.
You can even make a work around for the (current) lack of paging when return an ObjectSet to Slate by using the last value from the property ordered on (i.e. date_of_issue) as a filter in the subsequent request and return the next N objects.
This can work if you need a Slate table or HTML widget that renders on set of results then, on a user action, gets the next page.
I'm implementing a searchbar in IONIC 2 that search a JSON in one view so it can send its details to another view.
I have this JSON:
{
"Alphaville I": { //FIRST KEY
"ida": [{ //SECOND KEYS
"hora": "05:40",
"local": "AV. FERNÃO DIAS PAES LEME (Pref. Várzea Paulista)"
},... ],
"volta": [{ //SECOND KEYS
"hora": "05:40",
"local": "AV. FERNÃO DIAS PAES LEME (Pref. Várzea Paulista)"
},... ]
}, ... //MULTIPLE ITENS
}
So, in one view i create a list with the first keys (like Alphaville I), but i need to search the local inside of it.
But the Angular 2 *ngFor requires an array, so i iterate through my object and push it to an array, doing this it excludes my first key, so what i'm doing now (without searching, of course) is saving the keys in one array, geting the index and passing the jsonResultExample[index] to another page.
i'm using the basic searchbar example like the one in Seachbar Component Docs.
So what i need is: Search by the local key and return the first key (Alphaville I) of the nodes the contain the input text, the same local can appear in other first keys.
How can i do this? I can't post a better code because i haven't tried anything.
Is there a better way to structure my JSON for this? (i'm using firebase btw);
Any help or ideas is welcome, thanks.
EDIT
So i saved the first key value along with ida and volta so i can simply iterate through it, get the key value and everything without many problems, but since i need to filter by local it appears inside idaand volta as another array (cause i have many of these values), so it's looking like this now:
So now how can i access the local? Is it better to create another object only with all local and a key for every linha so i can return the values?
Remembering this is the searchbar code for Ionic 2 and my JSON has over 4k lines:
getItems(ev: any) {
// Reset items back to all of the items
this.initializeItems();
// set val to the value of the searchbar
let val = ev.target.value;
// if the value is an empty string don't filter the items
if (val && val.trim() != '') {
this.items = this.items.filter((item) => {
return (item.toLowerCase().indexOf(val.toLowerCase()) > -1);
})
}
}
Thanks in advance :)
It comes to personal experience on how to design data structure. Therefore I can't say the follow method is the best way.
First, in the case that we have complicated data structure, I don't prefer using map (a.k.a. object as data structure) in javascript. The main reason is pretty related to what you are facing, object by design cannot be iterated. Yes you can use Object.keys() or Object.values() but they are so ugly and hard to fit on every cases.
It is a nice move to put your first key as a property. That comes to the second problem. There seems to be an assumption in your structure that, one linha is mapped only to one local or one local is only related to one linha. If so, I suggest building another separated map only for the linha and local relationship.
Another approach is to normalize your data structure in to multiple separated javascript objects like what you do on database. By doing so, you can maximize the data flexibility that you can query whatever you want by Array.prototype.filter(), Array.prototype.map() or even directly access by its index. However, this approach may increase the lines of code as you need to manage multiple maps.
I use Couchbase Java SDK 2.2.6 with Couchbase server 4.1.
I query my view with the following code
public <T> List<T> findDocuments(ViewQuery query, String bucketAlias, Class<T> clazz) {
// We specifically set reduce false and include docs to retrieve docs
query.reduce(false).includeDocs();
log.debug("Find all documents, query = {}", decode(query));
return getBucket(bucketAlias)
.query(query)
.allRows()
.stream()
.map(row -> fromJsonDocument(row.document(), clazz))
.collect(Collectors.toList());
}
private static <A> A fromJsonDocument(JsonDocument saved, Class<A> clazz) {
log.debug("Retrieved json document -> {}", saved);
A object = fromJson(saved.content(), clazz);
return object;
}
In the logs from the fromJsonDocument method I see that rows are not always sorted by the row key. Usually they are, but sometimes they are not.
If I just run this query in browser couchbase GUI, I always receive results in expected order. Is it a bug or expected that view query results are not sorted when queried with async client?
What is the behaviour in different clients, not java?
This is due to the asynchronous nature of your call in the Java client + the fact that you used includeDocs.
What includeDocs will do is that it will weave in a call to get for each document id received from the view. So when you look at the asynchronous sequence of AsyncViewRow with includeDocs, you're actually looking at a composition of a row returned by the view and an asynchronous retrieval of the whole document.
If a document retrieval has a little bit of latency compared to the one for the previous row, it could reorder the (row+doc) emission.
But good news everyone! There is a includeDocsOrdered alternative in the ViewQuery that takes exactly the same parameters as includeDocs but will ensure that AsyncViewRow come in the same order returned by the view.
This is done by eagerly triggering the get retrievals but then buffering those that arrive out of order, so as to maintain the original order without sacrificing too much performance.
That is quite specific to the Java client, with its usage of RxJava. I'm not even sure other clients have the notion of includeDocs...
How can i write condition in Couchbase views to filter deleted collection from result.
function (doc, meta) {
if(doc.type=='folder'){
emit(doc.folder_id, doc);
}
}
Here is my simple view, It gives some deleted folders in json response.
Keep in mind that views are only eventually-consistent. They dont return real-time data. It can take a little time for views to actually reflect the data in CB.
Specifically, it takes up to a minute for deleted items to get removed from CB view results. This is somewhat configurable.
Important : dont emit "doc" in your views - that's wasteful (since it stores another copy of each document in CB).
Instead, you can usually request that the document is attached to
the result (in python its called "include_doc" - http://pythonhosted.org/couchbase/api/views.html). This can help you with the problem at hand: if you request that documents are included with the view results, then if the document has been deleted, it would simply not return it - telling you that its already been deleted.
Alternatively, if you dont have 'include_doc' in your client, you could do it manually: get the view results (just the list of document ids) and for each document, perform "get" to retrieve the data. if you can't find the doc, its been deleted. (unlike views, 'get' actions are fully consistent - they reflect the data that was written to disc).
I'm designing my back-end. I have a json array/queue/something which I only need any data that is at most 2 weeks old, that is continuously appended to. I only want to delete from this "queue", but not the container document. Can I use TTL for this, or does TTL only work for whole documents?
Is there a better way to do this? Should I store them in per-day or per-hour arrays as separate documents instead?
Running couchbase 2.2.
TTL in Couchbase only applies to whole documents, it's not possible to expire subsets of a document. Like you said you can always have separate documents with different expiry times in which you have a type,date and then the array of data as an element.
Then using a view like so:
function (doc, meta) {
if(meta.type == "json") {
if(doc.type == "ordered_data") {
if(doc.date) {
emit(dateToArray(doc.date));
}
}
}
}
You could emit all the related data ordered by date (flag descending set to true), it'd also allow your app to select specific dates by passing in one or more keys. I.e. selecting a date range of 2days,1week etc. When the document expires it'd be removed from the view when it updates (varies based upon your stale parameters plus ops a second/time).
Then you can do whatever joining or extra processing you need at the application layer. There are other options available but for me this would be the most sensible way to approach the problem, any problems just comment and we'll try again.
P.s. How big are you arrays going to become? If they are going to be very large then perhaps you'd need to look at a different tech or way to solve the problem.