Query is being cached - mysql

Hi guys I'm having a strange problem, and I'm looking for help.
I'm using laravel 7, and one of my queries apparently is being cached or something, here's my code to explain:
Here's a very simple method in one of my controllers:
public function chartEstadosVoluntarios(){
$estados = DB::table("voluntarios")
->selectRaw("estado, count(*) as total")
->whereNotNull('estado')
->groupBy("estado")
->get();
return response()->json($estados,200);
}
If I hit the route on chrome the result is:
[
{
"estado": "Derivado al 147",
"total": 2
},
{
"estado": "Derivar a mesa de ayuda",
"total": 7
},
{
"estado": "Pendiente",
"total": 3
},
{
"estado": "Reasignar Adulto Mayor",
"total": 50
},
{
"estado": "Resolución logística 24hs",
"total": 4
},
{
"estado": "Resuelto",
"total": 119
}
]
But its wrong, its not the real result, here's a screen for the same query in MySQL Workbench:
I'm using the same user in workbench and laravel.
Another thing is that If I slightly modify the query, then i get the correct result, for example:
$estados = DB::table("voluntarios")
->selectRaw("estado, count(*) as total, 'test'")//adding 'test' to select
->whereNotNull('estado')
->groupBy("estado")
->get();
Then I hit the browser again, and the result is:
[
{
"estado": "Pendiente",
"total": 9687,
"test": "test"
},
{
"estado": "Resuelto",
"total": 16415,
"test": "test"
}
]
If i go back with the modifications i get the wrong results again.
So, I really don't understand what's happening, Any point of direction will be appreciated.
EDIT, more info added from the answers:
the toSql() method throws the same query (copy pasted from the browser):
"select estado, count(*) as total from `voluntarios` where `estado` is not null group by `estado`";
The toSql() method with 'test' in select:
"select estado, count(*) as total, 'test' from `voluntarios` where `estado` is not null group by `estado`"
Same results in Tinker:
Downgraded from laravel v7.10.3 to v7.6.2 (the one that i have before running composer update today), but the problem remains.
EDIT 2
Weird thing happens, I recently use another part of the system that edits the "estado" field of table "voluntarios" to see what happen, and now the query throws the correct results. it's like something has been unlocked after editing the table.
What could it be? I check before that the user had no open transactions.
Thank's in advance.

Related

Laravel Eloquent: with() vs join() JSON Output

I have these 2 queries on my Laravel project:
$questions = \App\Answer::rightJoin('questions','answers.id_question','=','questions.id_question')
->leftJoin('users','answers.id_user','=','users.id')->where([
['questions.id_question', '=', $id_question],
['questions.flg_active', '=', true],
])->orderBy('questions.created_at', 'desc')->paginate(5);
and
$questions = \App\Answer::with('question.user')
->orderBy('created_at', 'desc')->paginate(15);
Essencially the first on returns questions not yet answered and the second only returns questions already answered.
I`m trying to use the same blade view for both, but they have different JSON output structure.
The first one returns an one level JSON, while the second returns 3 levels of data.
"data": [{
"id_answer": 42,
"created_at": "2018-11-28 17:52:18",
"updated_at": "2019-05-24 15:09:14",
"id_user": 2,
"id_question": 42,
"str_answer": "onono onooono nonono onn ",
"str_question": "ononon ononon onononn ?",
"url_link": null,
"flg_active": 1,
"id": 1,
"name": "Paul",
"surname": null,
"email": "p#yahoo.com",
"certification": "CFA ...",
"about": "CS, ....."
}
and:
"data": [{
"id_answer": 64,
"created_at": "2019-05-16 15:46:54",
"updated_at": "2019-05-16 15:46:54",
"id_user": 2,
"id_question": 98,
"str_answer": "onono non ooonononn",
"question": {
"id_question": 98,
"created_at": "2019-05-16 15:06:31",
"updated_at": "2019-05-16 15:24:43",
"id_user": 2,
"str_question": "onono onnon ononon?",
"url_link": null,
"flg_active": 1,
"user": {
"id": 2,
"name": "Paul",
"surname": null,
"email": "p#outlook.com",
"created_at": "2018-12-06 05:50:09",
"updated_at": "2019-05-24 11:23:32",
"certification": null,
"about": null
}
}
}
How can I make the first one return in the same 3 levels as the second query does ?
Why this occurs?
The first one is leveraging query builder more and will execute one query which will look something like this:
SELECT * FROM answer
RIGHT JOIN questions ON answers.id_question = questions.id_question
LEFT JOIN users ON answers.id_user = users.id
WHERE questions.id_question = $id_question
AND questions.flg_active = true
ORDER BY questions.created_at desc
OFFSET 0
LIMIT 15
This query returns data from all the tables exactly how sql calculates it. Laravel / Eloquent has no idea how to reliably put this into an eloquent format.
The 2nd one will execute 3 queries which will look something like this:
SELECT * FROM answer
ORDER BY created_at desc
OFFSET 0
LIMIT 15
SELECT * FROM question
WHERE answer_Id IN ($answer_ids) // Will be like (1, 5, 6, 7). All the ids retrieve from the first query
SELECT * FROM user
WHERE queston_id IN ($user_ids)
Eloquent does 3 queries therefore it can create the structure you outlined reliably.
Ultimately it's a trade off, the first option is faster but the response is harder to work with in php which creates less readable code.
I recommend you look at the queries that are executed by each by using the built in query log.
\DB::enableQueryLog();
$builtQuery->get()
dd(\DB::getQueryLog());
This occurs because in your second query you utilize eager loading for your Eloquent relationships. Eloquent model's default toArray() method merges both it's attributes and loaded relationships into one representation.
On the other hand, your first query joins tables, but doesn't load any relationships. So the most straight-forward way you can go about this is to eager load those relationships in your first query just as you do in your second one:
$questions = \App\Answer::with('questions.user')
->rightJoin('questions','answers.id_question','=','questions.id_question')
->leftJoin('users','answers.id_user','=','users.id')->where([
['questions.id_question', '=', $id_question],
['questions.flg_active', '=', true],
])->orderBy('questions.created_at', 'desc')->paginate(5);
Note that this will produce extra SELECT ... WHERE id IN (...) queries for both Question and User models to be eager loaded into their respective relationships.

Aggregated distinct from find results

First of all db.collection.distinct("name"); gives not bad result for me, but the problem is that distinct has limitation (results must not be larger than the maximum BSON size), and I need to aggregate through it, right?
Another things is that I really want to do distinct from find filtered results, so from something like this:
db.collection.find({ name: { $exists: true, $ne: null }, state: "published" });
So the main idea is to save all published "name" values from full collection without any limitation in the json file.
So I used:
>cat 1.json
db.collection.distinct("name");
mongo db < 1.json > 2.json
Correct query is:
db.collection.aggregate([{$match:{name: { $exists: true, $ne: null },
state: "published" }},{$group: {_id: null, uniqueValues: {$addToSet:
"$name"}}}]);

Formatting Mysql returned Array for JSON

I am doing a mysql query that pulls relative data from 3 tables based on an ID provided. My problem comes when I try to format the sql query so that when I turn the resulting array into a JSON object the data is nested correctly.
My SQL Query as of now:
select * from Players join SchoolData on Players.School = SchoolData.ID join LocationData on SchoolData.Location = LocationData.ID where Players.ID =%d
When I JSONify the results of the above query I get:
{
"data": [
{
"City": "San Diego",
"Height": 77,
"ID": 7,
"Location": 7,
"Mascot": "Aztecs",
"Name": "San Diego State Aztecs",
"School": 11,
"State": "CA",
"Weight": 196
}
]
}
I am trying to get my results to look like the following example which nest the data that is pulled from other tables.
{
“data”:[
{
“student”:
{
“id”:xxx-xxx-xxx,
“name”: “name1”,
“school”:
{
“id”:xxxx,
“address”:
{
“id”:xxx,
“city”: “mycity”,
“state”: “OH”
},
“name”: “myschool”,
“mascot”: “wildcats”
},
“height”: 72,
“weight”: 180
},
]
}
I have been looking at the FOR JSON command (that is used in SQL) and instead of selecting * all I can use the AS command in my query which should help with my label before the value returned but not the nesting of data.
I am just not sure if this is done pre query or post.
UPDATE
I am finding the following mysql functions: https://dev.mysql.com/doc/refman/5.7/en/json-creation-functions.html but I am not sure if this is what I am looking for.
SELECT
std.id, std.name, sch.id, add.id, add.city, add.state, sch.name, sch.mascot, std.height, std.weight
FROM
student std LEFT JOIN school sch ON < join condition >
LEFT JOIN address ON < join condition >
FOR JSON AUTO
Try something like this, FOR JSON AUTO should convert it into JSON string that you want.

N1QL nested json, query on field inside object inside array

I have json documents in my Couchbase cluster that looks like this
{
"giata_properties": {
"propertyCodes": {
"provider": [
{
"code": [
{
"value": [
{
"name": "Country Code",
"value": "EG"
},
{
"name": "City Code",
"value": "HRG"
},
{
"name": "Hotel Code",
"value": "91U"
}
]
}
],
"providerCode": "gta",
"providerType": "gds"
},
{
"code": [
{
"value": [
{
"value": "071801"
}
]
},
{
"value": [
{
"value": "766344"
}
]
}
],
"providerCode": "restel",
"providerType": "gds"
},
{
"code": [
{
"value": [
{
"value": "HRG03Z"
}
]
},
{
"value": [
{
"value": "HRG04Z"
}
]
}
],
"providerCode": "5VF",
"providerType": "tourOperator"
}
]
}
}
}
I'm trying to create a query that fetches a single document based on the value of giata_properties.propertyCodes.provider.code.value.value and a specific providerType.
So for example, my input is 071801 and restel, I want a query that will fetch me the document I pasted above (because it contains these values).
I'm pretty new to N1QL so what I tried so far is (without the providerType input)
SELECT * FROM giata_properties AS gp
WHERE ANY `field` IN `gp.propertyCodes.provider.code.value` SATISFIES `field.value` = '071801' END;
This returns me an empty result set. I'm probably doing all of this wrongly.
edit1:
According to geraldss answer I was able to achieve my goal via 2 different queries
1st (More general) ~2m50.9903732s
SELECT * FROM giata_properties AS gp WHERE ANY v WITHIN gp SATISFIES v.`value` = '071801' END;
2nd (More specific) ~2m31.3660388s
SELECT * FROM giata_properties AS gp WHERE ANY v WITHIN gp.propertyCodes.provider[*].code SATISFIES v.`value` = '071801' END;
Bucket have around 550K documents. No indexes but the primary currently.
Question part 2
When I do either of the above queries, I get a result streamed to my shell very quickly, then I spend the rest of the query time waiting for the engine to finish iterating over all documents. I'm sure that I'll be only getting 1 result from future queries so I thought I can use LIMIT 1 so the engine stops searching on first result, I tried something like
SELECT * FROM giata_properties AS gp WHERE ANY v WITHIN gp SATISFIES v.`value` = '071801' END LIMIT 1;
But that made no difference, I get a document written to my shell and then keep waiting until the query finishes completely. How can this be configured correctly?
edit2:
I've upgraded to the latest enterprise 4.5.1-2844, I have only the primary index created on giata_properties bucket, when I execute the query along with the LIMIT 1 keyword it still takes the same time, it doesn't stop quicker.
I've also tried creating the array index you suggested but the query is not using the index and it keeps insisting on using the #primary index (even if I use USE INDEX clause).
I tried removing SELF from the index you suggested and it took a much longer time to build and now the query can use this new index, but I'm honestly not sure what I'm doing here.
So 3 questions:
1) Why LIMIT 1 using primary index doesn't make the query stop at first result?
2) What's the difference between the index you suggested with and without SELF? I tried to look for SELF keyword documentation but I couldn't find anything.
This is how both indexes look in Web ui
Index 1 (Your original suggestion) - Not working
CREATE INDEX `gp_idx1` ON `giata_properties`((distinct (array (`v`.`value`) for `v` within (array_star((((self.`giata_properties`).`propertyCodes`).`provider`)).`code`) end)))
Index 2 (Without SELF)
CREATE INDEX `gp_idx2` ON `giata_properties`((distinct (array (`v`.`value`) for `v` within (array_star(((self.`propertyCodes`).`provider`)).`code`) end)))
3) What would be the query for a specific giata_properties.propertyCodes.provider.code.value.value and a specific providerCode? I managed to do both separately but I wasn't successful in merging them.
Thanks for all your help dear
Here is a query without the providerType.
EXPLAIN SELECT *
FROM giata_properties AS gp
WHERE ANY v WITHIN gp.giata_properties.propertyCodes.provider[*].code SATISFIES v.`value` = '071801' END;
You can also index this in Couchbase 4.5.0 and above.
CREATE INDEX idx1 ON giata_properties( DISTINCT ARRAY v.`value` FOR v WITHIN SELF.giata_properties.propertyCodes.provider[*].code END );
Edit to answer question edits
The performance has been addressed in 4.5.x. You should try the following on Couchbase 4.5.1 and post the execution times here.
Test on 4.5.1.
Create the index.
Use the LIMIT. In 4.5.1, the limit is pushed down to the index.

Using addToSet inside an array with MongoDB

I'm trying to track daily stats for an individual.
I'm having a hard time adding a new day inside "history" and can also use a pointer on updating "walkingSteps" as new data comes in.
My schema looks like:
{
"_id": {
"$oid": "50db246ce4b0fe4923f08e48"
},
"history": [
{
"_id": {
"$oid": "50db2316e4b0fe4923f08e12"
},
"date": {
"$date": "2012-12-24T15:26:15.321Z"
},
"walkingSteps": 10,
"goalStatus": 1
},
{
"_id": {
"$oid": "50db2316e4b0fe4923f08e13"
},
"date": {
"$date": "2012-12-25T15:26:15.321Z"
},
"walkingSteps": 5,
"goalStatus": 0
},
{
"_id": {
"$oid": "50db2316e4b0fe4923f08e14"
},
"date": {
"$date": "2012-12-26T15:26:15.321Z"
},
"walkingSteps": 8,
"goalStatus": 0
}
]
}
db.history.update( ? )
I've been browsing (and attempting) the mongodb documentation but they don't quite break it all the way down to dummies like myself... I couldn't quite translate their examples to my setup.
Thanks for any help.
E = noob trying to learn programming
Adding a day:
user = {_id: ObjectId("50db246ce4b0fe4923f08e48")}
day = {_id: ObjectId(), date: ISODate("2013-01-07"), walkingSteps:0, goalStatus: 0}
db.users.update(user, {$addToSet: {history:day}})
Updating walkingSteps:
user = ObjectId("50db246ce4b0fe4923f08e48")
day = ObjectId("50db2316e4b0fe4923f08e13") // second day in your example
query = {_id: user, 'history._id': day}
db.users.update(query, {$set: {"history.$.walkingSteps": 6}})
This uses the $ positional operator.
It might be easier to have a separate history collection though.
[Edit] On the separate collections:
Adding days grows the document in size and it might need to be relocated on the disk. This can lead to performance issues and fragmentation.
Deleting days won't shrink the document size on disk.
It makes querying easier/straightforward (e.g. searching for a period of time)
Even though #Justin Case puts the right answer he doesn't explain a few things in it extremely well.
You will notice first of all that he gets rid of the resolution on dates and moves their format to merely the date instead of date and time like so:
day = {_id: ObjectId(), date: ISODate("2013-01-07"), walkingSteps:0, goalStatus: 0}
This means that all your dates will have 00:00:00 for their time instead of the exact time you are using atm. This increases the ease of querying per day so you can do something like:
db.col.update(
{"_id": ObjectId("50db246ce4b0fe4923f08e48"),
"history.date": ISODate("2013-01-07")},
{$inc: {"history.$.walkingSteps":0}}
)
and other similar queries.
This also makes $addToSet actually enforce its rules, however since the data in this sub document could change, i.e. walkingSteps will increment $addToSet will not work well here anyway.
This is something I would change from the ticked answer. I would probably use $push or something else instead since $addToSet is heavier and won't really do anything useful here.
The reason for a separate history collection in my view would be what you said earlier with:
Yes, the amount of history items for that day.
So this array contains a set of days, which is fine but it sounds like the figure that you wish to get walkingSteps from, a set of history items, should be in another collection and you set walkingSteps according to the count of the amount of items in that other collection for today:
db.history_items.find({date: ISODate("2013-01-07")}).count();
Referring to MongoDB Manual, $ is the positional operator which identifies an element in an array field to update without explicitly specifying the position of the element in the array. The positional $ operator, when used with the update() method and acts as a placeholder for the first match of the update query selector.
So, if you issue a command to update your collection like this:
db.history.update(
{someCriterion: someValue },
{ $push: { "history":
{"_id": {
"$oid": "50db2316e4b0fe4923f08e12"
},
"date": {
"$date": "2012-12-24T15:26:15.321Z"
},
"walkingSteps": 10,
"goalStatus": 1
}
}
)
Mongodb might try to identify $oid and $date as some positional parameters. $ also is part of the atomic operators like $set and $push. So, it is better to avoid use this special character in Mongodb.