I want to retrieve data from a Couchbase bucket with N1QL between two days (from 00:00:00.000 start date time until 23:59:59.999 end date time)
Is it enough to provide the dates without time, like this:
SELECT c.*
FROM customer c
WHERE c.start BETWEEN '2017-10-09' AND '2017-10-10'
Or do I need to provide the time explicitly:
SELECT c.*
FROM customer c
WHERE c.start BETWEEN '2017-10-09T00:00:00.000Z' AND '2017-10-10T23:59:59.999Z')
?
Couchbase date in ISO-8601 are string comparable. The both queries are fine. Checkout more details https://developer.couchbase.com/documentation/server/current/n1ql/n1ql-language-reference/datefun.html
You need to use a view with a map function that convert date to Array like this :
function (doc, meta) {
emit(dateToArray(doc.start), doc);
}
and then request the view with the startKey and endKey filter
Exemple with NodeJS client use the range method:
const viewQuery = couchbase.ViewQuery.from(nameOfYourDesignDocument, nameOfYourView);
bucket.query(viewQuery.range([2017,10,9], [2017,10,10]), (error, rows) => {
});
Related
Let me preface by saying I'm very new to SQL (and back end design) in general. So for those annoyed with noob questions, please be gentle.
BACKGROUND:
I'm trying to build a product test database (storing test data for all our products) where I want a user to be able to refine a search to find test data they actually want. For example, they may start by searching for all products of a certain brand name, and then refine it with a product type, and/or refine it with a date range of when the test was done.
PROBLEM:
I'm having a hard time finding information on how to implement multi-parameter searches with mysql and node.js. I know you can do nested queries and joins and such within pure SQL syntax, but it's not abundantly clear to me how I would do this from node.js, especially when certain search criteria aren't guaranteed to be used.
Ex:
CREATE PROCEDURE `procedureName`(
IN brandname VARCHAR(20),
producttype VARCHAR(30))
BEGIN
SELECT * FROM products
WHERE brand = brandname
AND product_type = producttype;
END
I know how to pass data from node.js to this procedure, but what if the user didn't specify a product type? Is there a way to nullify this part of the query? Something like:
AND product_type = ALL;
WHAT I'VE TRIED:
I've also looked into nesting multiple SQL procedures, but passing in dynamic data to the "FROM" clause doesn't seem to be possible. Ex: if I had a brandname procedure, and a product type procedure, I don't know how/if I can pass the results from one procedure to the "FROM" clause of the other to actually refine the search.
One idea was to create tables with the results in each of these procedures, and pass those new table names to subsequent procedures, but that strikes me as an inefficient way to do this (Am I wrong? Is this a completely legit way to do this?).
I'm also looking into building a query string on the node side that would intelligently decide what search criteria have been specified by the front end, and figure out where to put SQL AND's and JOIN's and what-nots. The example below actually works, but this seems like it could get ugly quick as I add more search criteria, along with JOINS to other tables.
// Build a SQL query based on the parameters in a request URL
// Example request URL: http://localhost:3000/search?brand=brandName&type=productType
function qParams(req) {
let q = "SELECT * FROM products WHERE ";
let insert = [];
if(req.query.brand) {
brandname = req.query.brand; // get brandname from url request
q = q + `brand = ?`, // Build brandname part of WHERE clause
insert.push(brandname); // Add brandname to insert array to be used with query.
};
if(req.query.type) {
productType = req.query.type; // get product type from url request
insert.length > 0 ? q = q + ' AND ' : q = q; // Decide if this is the first search criteria, add AND if not.
q = q + 'product_type = ?'; // Add product_type to WHERE clause
insert.push(productType); // Add product_type variable to insert array.
}
// Return query string and variable insert array
return {
q: q,
insert: insert
};
};
// Send Query
async function qSend(req, res) {
const results = await qParams(req); // Call above function, wait for results
// Send query string and variables to MySQL, send response to browser.
con.query(results.q, results.insert, (err, rows) => {
if(err) throw err;
res.send(rows);
res.end;
})
};
// Handle GET request
router.use('/search', qSend);
CONCISE QUESTIONS:
Can I build 1 SQL procedure with all my search criteria as variables, and nullify those variables from node.js if certain criteria aren't used?
Is there way to nest multiple MySQL procedures so I can pick the procedures applicable to the search criteria?
Is creating tables of results in a procedure, and passing those new table names to other procedures a reasonable way to do that?
Building the query from scratch in node is working, but it seems bloated. Is there a better way to do this?
Googling "multi-parameter search mysql nodejs" is not producing useful results for my question, i.e. I'm not asking the right question. What is the right question? What do I need to be researching?
One option is to use coalesce():
SELECT p.*
FROM products p
WHERE
p.brand = COALESCE(:brandname, p.brand)
AND p.product_type = COALESCE(:producttype, p.producttype);
It may be more efficient do explicit null checks on the parameters:
SELECT p.*
FROM products p
WHERE
(:brandname IS NULL OR p.brand = :brandname)
AND (:producttype IS NULL OR p.product_type = :producttype);
I'd like to make a query on dates using SequelizeJS but i don't know how to do and there is nothing on that on the website...
My code :
var day = request.params.day;
Appointment.findAll({where: ["start.day() = day"]}); // start is my column, format with DATETIME
Depending on your DB there might be some function to extract the day from the column. Never seen the .day syntax before though.
Appointment.findAll({
where: sequelize.where(sequelize.fn('day', sequelize.col('start')), day)
});
On latest master this should produce something like
WHERE DAY("start") = day
You can just use the regular comparison operator, as long as your date is in a valid format. For this purpose you can just create a Date object out of the input, and then pass it to the Sequelize query, like this:
var day = new Date(request.params.day);
Appointment.findAll({where: ['start > ?', day]}).then(function (result) {
// do stuff here
});
Currently when I need to run a query that will be used w/ paging I do it something like this:
//Setup query (Typically much more complex)
var q = ctx.People.Where(p=>p.Name.StartsWith("A"));
//Get total result count prior to sorting
int total = q.Count();
//Apply sort to query
q = q.OrderBy(p => p.Name);
q.Select(p => new PersonResult
{
Name = p.Name
}.Skip(skipRows).Take(pageSize).ToArray();
This works, but I wondered if it is possible to improve this to be more efficient while still using linq? I couldn't think of a way to combine the count w/ the data retrieval in a single trip to the DB w/o using a stored proc.
The following query will get the count and page results in one trip to the database, but if you check the SQL in LINQPad, you'll see that it's not very pretty. I can only imagine what it would look like for a more complex query.
var query = ctx.People.Where (p => p.Name.StartsWith("A"));
var page = query.OrderBy (p => p.Name)
.Select (p => new PersonResult { Name = p.Name } )
.Skip(skipRows).Take(pageSize)
.GroupBy (p => new { Total = query.Count() })
.First();
int total = page.Key.Total;
var people = page.Select(p => p);
For a simple query like this, you could probably use either method (2 trips to the database, or using GroupBy to do it in 1 trip) and not notice much difference. For anything complex, I think a stored procedure would be the best solution.
Jeff Ogata's answer can be optimized a little bit.
var results = query.OrderBy(p => p.Name)
.Select(p => new
{
Person = new PersonResult { Name = p.Name },
TotalCount = query.Count()
})
.Skip(skipRows).Take(pageSize)
.ToArray(); // query is executed once, here
var totalCount = results.First().TotalCount;
var people = results.Select(r => r.Person).ToArray();
This does pretty much the same thing except it won't bother the database with an unnecessary GROUP BY. When you are not certain your query will contain at least one result, and don't want it to ever throw an exception, you can get totalCount in the following (albeit less cleaner) way:
var totalCount = results.FirstOrDefault()?.TotalCount ?? query.Count();
Important Note for People using EF Core >= 1.1.x && < 3.0.0:
At the time I was looking for solution to this and this page is/was Rank 1 for the google term "EF Core Paging Total Count".
Having checked the SQL profiler I have found EF generates a SELECT COUNT(*) for every row that is returned. I have tired every solution provided on this page.
This was tested using EF Core 2.1.4 & SQL Server 2014. In the end I had to perform them as two separate queries like so. Which, for me at least, isn't the end of the world.
var query = _db.Foo.AsQueryable(); // Add Where Filters Here.
var resultsTask = query.OrderBy(p => p.ID).Skip(request.Offset).Take(request.Limit).ToArrayAsync();
var countTask = query.CountAsync();
await Task.WhenAll(resultsTask, countTask);
return new Result()
{
TotalCount = await countTask,
Data = await resultsTask,
Limit = request.Limit,
Offset = request.Offset
};
It looks like the EF Core team are aware of this:
https://github.com/aspnet/EntityFrameworkCore/issues/13739
https://github.com/aspnet/EntityFrameworkCore/issues/11186
I suggest making two queries for the first page, one for the total count and one for the first page or results.
Cache the total count for use as you move beyond the first page.
Using the Stripe API, I'd like to be able to query for date ranges or, failing that, dates that are greater than or less than some arbitrary date.
I know that I can query for something based on an exact date, e.g.,:
https://api.stripe.com/v1/events?created=1336503409
But I want something along the lines of...
# Search for events where `created` is greater than the epoch time: 1336503400
https://api.stripe.com/v1/events?created__gt=1336503400
Yes. From https://stripe.com/docs/api?lang=curl#list_events
created: A filter on the list based on the events created date. The value can be a string with an exact UTC timestamp, or it can be a dictionary with the following options:
gt (optional) Return values should have been created after this timestamp.
gte (optional) Return values should have been created after or equal to this timestamp.
lt (optional) Return values should have been created before this timestamp.
lte (optional) Return values should have been created before or equal to this timestamp.
So with curl, you can build a request like this:
curl "https://api.stripe.com/v1/events?created%5Blt%5D=1337923293" \
-u ${YOUR_API_KEY}:
The unescaped query parameter is created[lt]=1337923293.
If you're looking how to do so with the ruby client, here it is:
Stripe::Charge.all(limit: 100, 'created[lt]' => timestamps })
Along the same lines. You can achieve the same thing with the Python client like so:
import stripe
from datetime import datetime, timedelta
my_date = '1/31/2011'
my_timestamp = datetime.strptime(my_date, "%m/%d/%Y").timestamp()
stripe.Charge.all(created={'lt': my_timestamp})
For those of us using the Go Api, both formats below works for selecting a range:
params := &stripe.ChargeListParams{}
params.Filters.AddFilter("created", "lt", strconv.Itoa(1592217599))
params.Filters.AddFilter("created", "gt", strconv.Itoa(1591612124))
i := charge.List(params)
params := &stripe.ChargeListParams{}
params.Filters.AddFilter("created[lt]", "", strconv.Itoa(1592217599))
params.Filters.AddFilter("created[gt]", "", strconv.Itoa(1591612124))
i := charge.List(params)
I've stumbled upon a very strange LINQ to SQL behaviour / bug, that I just can't understand.
Let's take the following tables as an example: Customers -> Orders -> Details.
Each table is a subtable of the previous table, with a regular Primary-Foreign key relationship (1 to many).
If I execute the follow query:
var q = from c in context.Customers
select (c.Orders.FirstOrDefault() ?? new Order()).Details.Count();
Then I get an exception: Could not format node 'Value' for execution as SQL.
But the following queries do not throw an exception:
var q = from c in context.Customers
select (c.Orders.FirstOrDefault() ?? new Order()).OrderDateTime;
var q = from c in context.Customers
select (new Order()).Details.Count();
If I change my primary query as follows, I don't get an exception:
var q = from r in context.Customers.ToList()
select (c.Orders.FirstOrDefault() ?? new Order()).Details.Count();
Now I could understand that the last query works, because of the following logic:
Since there is no mapping of "new Order()" to SQL (I'm guessing here), I need to work on a local list instead.
But what I can't understand is why do the other two queries work?!?
I could potentially accept working with the "local" version of context.Customers.ToList(), but how to speed up the query?
For instance in the last query example, I'm pretty sure that each select will cause a new SQL query to be executed to retrieve the Orders. Now I could avoid lazy loading by using DataLoadOptions, but then I would be retrieving thousands of Order rows for no reason what so ever (I only need the first row)...
If I could execute the entire query in one SQL statement as I would like (my first query example), then the SQL engine itself would be smart enough to only retrieve one Order row for each Customer...
Is there perhaps a way to rewrite my original query in such a way that it will work as intended and be executed in one swoop by the SQL server?
EDIT:
(longer answer for Arturo)
The queries I provided are purely for example purposes. I know they are pointless in their own right, I just wanted to show a simplistic example.
The reason your example works is because you have avoided using "new Order()" all together. If I slightly modify your query to still use it, then I still get an exception:
var results = from e in (from c in db.Customers
select new { c.CustomerID, FirstOrder = c.Orders.FirstOrDefault() })
select new { e.CustomerID, Count = (e.FirstOrder != null ? e.FirstOrder : new Order()).Details().Count() }
Although this time the exception is slightly different - Could not format node 'ClientQuery' for execution as SQL.
If I use the ?? syntax instead of (x ? y : z) in that query, I get the same exception as I originaly got.
In my real-life query I don't need Count(), I need to select a couple of properties from the last table (which in my previous examples would be Details). Essentially I need to merge values of all the rows in each table. Inorder to give a more hefty example I'll first have to restate my tabels:
Models -> ModelCategoryVariations <- CategoryVariations -> CategoryVariationItems -> ModelModuleCategoryVariationItemAmounts -> ModelModuleCategoryVariationItemAmountValueChanges
The -> sign represents a 1 -> many relationship. Do notice that there is one sign that is the other way round...
My real query would go something like this:
var q = from m in context.Models
from mcv in m.ModelCategoryVariations
... // select some more tables
select new
{
ModelId = m.Id,
ModelName = m.Name,
CategoryVariationName = mcv.CategoryVariation.Name,
..., // values from other tables
Categories = (from cvi in mcv.CategoryVariation.CategoryVariationItems
let mmcvia = cvi.ModelModuleCategoryVariationItemAmounts.SingleOrDefault(mmcvia2 => mmcvia2.ModelModuleId == m.ModelModuleId) ?? new ModelModuleCategoryVariationItemAmount()
select new
{
cvi.Id,
Amount = (mmcvia.ModelModuleCategoryVariationItemAmountValueChanges.FirstOrDefault() ?? new ModelModuleCategoryVariationItemAmountValueChange()).Amount
... // select some more properties
}
}
This query blows up at the line let mmcvia =.
If I recall correctly, by using let mmcvia = new ModelModuleCategoryVariationItemAmount(), the query would blow up at the next ?? operand, which is at Amount =.
If I start the query with from m in context.Models.ToList() then everything works...
Why are you looking into only the individual count without selecting anything related to the customer.
You can do the following.
var results = from e in
(from c in db.Customers
select new { c.CustomerID, FirstOrder = c.Orders.FirstOrDefault() })
select new { e.CustomerID, DetailCount = e.FirstOrder != null ? e.FirstOrder.Details.Count() : 0 };
EDIT:
OK, I think you are over complicating your query.
The problem is that you are using the new WhateverObject() in your query, T-SQL doesnt know anyting about that; T-SQL knows about records in your hard drive, your are throwing something that doesn't exist. Only C# knows about that. DON'T USE new IN YOUR QUERIES OTHER THAN IN THE OUTER MOST SELECT STATEMENT because that is what C# will receive, and C# knows about creating new instances of objects.
Of course is going to work if you use ToList() method, but performance is affected because now you have your application host and sql server working together to give you the results and it might take many calls to your database instead of one.
Try this instead:
Categories = (from cvi in mcv.CategoryVariation.CategoryVariationItems
let mmcvia =
cvi.ModelModuleCategoryVariationItemAmounts.SingleOrDefault(
mmcvia2 => mmcvia2.ModelModuleId == m.ModelModuleId)
select new
{
cvi.Id,
Amount = mmcvia != null ?
(mmcvia.ModelModuleCategoryVariationItemAmountValueChanges.Select(
x => x.Amount).FirstOrDefault() : 0
... // select some more properties
}
Using the Select() method allows you to get the first Amount or its default value. I used "0" as an example only, I dont know what is your default value for Amount.