When I query some view in Couchbase I get the response that has following structure:
{
"total_rows":100,
"rows":[...]
}
'total_rows' is very useful property that I can use for paging.
But lets say I select only a subset of view using 'start_key' and 'end_key' and of course I don't know how big this subset will be. 'total_rows' is still the same number (as I understand it's just total of whole view). Is there any easy way to know how many rows was selected in subset?
You can use the in-built reduce function _count to get the total count of your query.
Just add _count as reduce function for your view. After that, you will need to make two calls to couchbase:
In one call, you'll set the query param reduce=true (along with either group=true or group_level=n, depending upon how you're sending your key(s)). This will give you the total count of your filtered rows.
In the other call, you'll disable the reduce function with reduce=false because you now need the actual rows.
You can find more details about map and reduce at http://docs.couchbase.com/admin/admin/Views/views-writing.html
You can just use an array count/total/length in whatever language you are using.
For example in PHP:
$result = $cb->view("dev_beer", "beer_by_name", array('startkey' => 'O', 'endkey'=>'P'));
echo "total = >>".count($result["rows"])
If you're actually wanting to paginate your data then you should use limit and skip:
http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-views-writing-querying-pagination.html
If you have to paginate the view in the efficient way, you actually don't need to specify both start and the end.
Generally it is possible to use startkey/startkey_id and limit. In this case the limit will tell you that the page won't be bigger than known size.
Both cases are described in CouchDB book: http://guide.couchdb.org/draft/recipes.html#pagination
Here is how it works:
Request rows_per_page + 1 rows from the view
Display rows_per_page rows, store + 1 row as next_startkey and next_startkey_docid
As page information, keep startkey and next_startkey
Use the next_* values to create the next link, and use the others to create the previous link
Related
What is the best approach for searching?What will be difference if i filter the all data in controller and get result, and use where query in model and get result ?Please suggest your opinion.
It depends on the complexity of your query..
You can try to mesure the time of processing by putting flags in your code between each steps in order to measure the time of processing.
Then you make one test of speed processing like:
print time_flag 1
var results_sql_processing = *complex query*
print time flag 2
var raw_results_script_processing = *dump query*
print time flag 3
var results_script_processing = *processing the data*
print time flag 4
and make sure results_script_processing == results_sql_processing.
You can also set different datasets size (limit 100, 500, 1000) and see how the difference evolves between both solutions
Also, i would recommend to take a look at the query builders you might find in many frameworks (I use laravel's query builder).
They are usually a very good compromise when the query isn't too complex (no data aggregation, complex concat,....), you can still use joins, union and many filters on them.
But in case you want to get a super complex pivot table for instance, just build a strong sql query and then fire it in your code!
I have some reporting methods throughout my app and in some cases I want to return the count (for a dashboard), and in others, return the full result set for viewing the details of a report.
What I'm wondering, is there a way to dynamically choose to show the count (instead of what I'm doing here):
def get_query_results(reporting_parameters, count_only = true)
#put together reporting details...
if count_only
MyModel.where(query).count
else
MyModel.where(query)
end
end
I considered setting a local variable to the result of my query parameter, and then call count, but that queries the database again (and even if it didn't it could increase memory usage).
Is there a way to do an effective way to do this in one query? This is one of several queries I have like this in my app, otherwise I wouldn't care. Also, I'd use a ternary, but the actual query conditions in my app are much longer than my example here and it makes it unreadable.
Suppose you are doing this:
#collection = get_query_results(...)
Then you can do this afterwards instead of inside of the action:
#collection.count
And if you like to call another method:
def total_number(collection)
collection.count
end
#collection = get_query_results(...)
no_of_records = total_number(#collection)
I am currently designing a Rest API and is a little stuck on performance matters for 2 of the use cases in the system:
List all campaigns (api/campaigns) - needs to return campaign data needed for listing and paging campaigns. Maybe return up to 1000 records and would take ages to retreive and return detailed data. The needed data can be returned in a single DB call.
Retrieve campaign item (api/campaigns/id) - need to return all data about the campaign and may take up to a second to run. Multiple DB calls is needed to get all campaign data for a single campaign.
My question is: Is it valid to return different json-responses to those 2 calls (if well documented) even if it regards the same resource? I am thinking that the list response is a sub set of the retreive-response. The reason for this is to make to save DB calls and bandwitdh + parsing.
Thanks in advance!
I think it's both fine and expected for /campaigns and /campaigns/{id} to return different information. I would suggest using query parameters to limit the amount of information you need to return. For instance, only return a URI to each player unless you see a ?expand=players query parameter, in which case you return detailed player information.
I am looking at this.
I have queries that return up to 10,000 rows and usually fills up the memory
I am currently doing:
$this->db->get()->result_array();
Is there a way to not load all of the data into memory and use some sort of cursor? It seems result and result array are both arrays. (one is array of objects other is an array of arrays)
If you are using Active Record, which I would personally recommend, limit() should achieve what you are looking for.
$this->db->limit();
Can also use in this way, slightly easier and less lines of code:
$query = $this->db->get('Table_Name', 50); //syntax db->get('Table',limit_val);
return $query->result();
Also can return a limit with an offset:
$this->db->limit(10, 20); // Second parameter lets you set a result offset
Link for more help on Active Record Query's
https://www.codeigniter.com/userguide2/database/active_record.html#select
Mysql offset is the pointer you are looking for.
You can use it like:
$this->db->get(tableName,10,20);
The second parameter and the third one help you to set limit and offset .
Here are more details
i'm working in a symfony project and using sfPropelPager to show a paged list of elements.
The problem is that with a great amount of data to list (i.e. thousands of registers) it makes a query to the database for each page to show!!!! That means about 100 extra queries in my case, and that is unacceptable.
Showing some of my code: the function that returns the pager object
$pager = new sfPropelPager('MyTable',sfConfig::get('sfPropelPagerLines'));
$c = new Criteria();
$c->add('my_table_field',$value);
$c->addDescendingOrderByColumn('date');
$pager->setCriteria($c);
$pager->init();
return $pager;
So, please, if you know a way to get all the results with only one query, it would be a great solution for my problem. Otherwise i must implement that list with an ajax call for every page the user wants to see
Thank you very much for your time.
I'm not sure to get your problem but, anyway, avoid the use of Criteria. Try to make queries with the ModelCriteria API: http://www.propelorm.org/reference/model-criteria.html.
For each paginated page, a query to the database will be done, this is the standard behavior for all pagers I know. If it's related to related objects (assuming you want to display information from relations), you may want to create a query that links those objects before to paginate, that way you'll get one query per page for all your data to display.
Read this doc for instance: http://www.propelorm.org/documentation/03-basic-crud.html#query_termination_methods
At last i did'nt get a solution for the problem, i had to implement the list via AJAX call, calling to a function that returns the requested page, so at the load of the page, no query for this list is slowing the user experience.
Thank you anyway to help me :)