Complex conditions on Eloquent Joins - mysql

Question

I'm looking to use Eloquent to output some data from our database, but the database can vary.
Below is an illustration of my database tables
Webservice Tags:
----------------------------------------------------------
| id | webservice_tag | webservice_name | blog_id |
==========================================================
| 1 | TEST, TESSST, TES | Test Service | 1 |
----------------------------------------------------------
| 2 | OPTION, OPT, EXAMPLE | Example Service | 1 |
----------------------------------------------------------
| 3 | ANOTHER, ANO, THER | Another Service | 1 |
----------------------------------------------------------
Blog Post:
----------------------------------------------------------
| id | title | blog_id | tag |
==========================================================
| 1 | Blog Title 1 | 1 | THER |
----------------------------------------------------------
| 2 | Blog Title 2 | 1 | TES |
----------------------------------------------------------
| 3 | Blog Title 3 | 1 | ANOTHER |
----------------------------------------------------------
So here, we have two tables. Blog Posts and Web-service Tags.
Our blog posts are populated by a number of web-services from a number of different providers. Test Service, Example Service and Another Service for example. However, these web-services are very inconsistent; they'll send over a combination of tags, and no two posts are guaranteed to be the same.
So, we've created a table called Webservice Tags which is designed to log each of these occurrances. This way we can identify that (in the examples) Blog Title 3 was sent by Another Service, Blog Title 2 was sent by Test Service etc.
I'm developing reports to show how many posts we get from each of our web services. So for each blog post, I need to identify the web service and get the web service name that's associated with it. We have multiple blogs, each with their own web services (some may share a tag), so this report needs to be isolated to each individual blog.
Here's the query in Eloquent:
$query = DB::table('blog_posts')
->join('webservice_tags', function($join) use ($blog) {
$join->on('blog_posts.tag', '=', 'webservice_tags.webservice_tag')
->where('webservice_tags.blog_id', '=', $blog->id);
})
->addSelect('webservice_tags.webservice_name AS name')
->addSelect(DB::raw("COUNT(blog_posts.id) AS count"))
->where('blog_posts.blog_id', '=', $blog->id)
->groupBy('webservice_tags.webservice_name')
->get();
This query was fine whilst the Webservice sent a consistent tag. Now however there are different tags per web service and this report needs to count them all equally.
Here's my amendment, but it's not working as expected:
->join('webservice_tags', function($join) use ($blog) {
$join->on('blog_posts.tag', 'LIKE', DB::raw('CONCAT("%", webservice_tags.webservice_tag, "%")'))
->where('webservice_tags.blog_id', '=', $blog->id);
})
I'm not getting any matches in the query through this method.
The query aims to join any field to the SELECT where the blog_posts.tag is within the comma separated list of webservice_tags.
Is there a way to do this more effectively?
Explanation on the blog_id relationship
This system manages multiple blogs, as well as the posts within those blogs. A webservice may be applicable to one blog, but not the other, which is why the blog_id is associated witih webservices as well as the individual posts. There are things like commission percentages that are associated with a webservice, and Test Webservice may offer 10% for Blog A, and 12% for Blog B, so they are essentially separate.

Answer 1

Quickest/crude solution:
The reason you do not see any result is because of this:
$join->on('blog_posts.tag', 'LIKE', DB::raw('CONCAT("%", webservice_tags.webservice_tag, "%")'))
which translates to:
INNER JOIN `webservice_tags`
ON `blog_posts`.`tag` LIKE CONCAT("%", webservice_tags.webservice_tag, "%")
This tries to match blog_posts.tag that has values like %ANOTHER, ANO, THER%.
So let's say blog_posts.tag is "ANOTHER" and webservice_tags.webservice_tag is "ANOTHER, ANO, THER". Rather than matching "ANOTHER", the database is trying to match values like these:
ANOTHER != FOOANOTHER, ANO, THER
ANOTHER != ANOTHER, ANO, THERBAR
ANOTHER != FOOANOTHER, ANO, THERBAR
Which is the reason why you are not getting any results. You will need to swap the columns around, like this:
$join->on('webservice_tags.webservice_tag', 'LIKE', DB::raw('CONCAT("%", blog_posts.tag, "%")'))
Longer solution:
I'm just going to throw my idea here. These are some points that come into my mind when thinking about this:
Tag <-> Web Service mapping is stored as a string when it is actually a set of data. That makes it hard to search and hard to perform queries on it. I like to break them down into two or more tables. This is called Database normalization. Don't ask me which level it is, I never understood the technical description for them. :D
Since you mentioned that a web service can have many tags, but some tags can also be shared by different web services, we will need a pivot table to handle this many-to-many relationship.
When aggregating, I like to start from top down. So I write queries starting from webservices, then work my way down to blog posts. This is the opposite of your example.
I'm ignoring blog_id columns in webservices table for now since I don't quite understand yet what they do.
So first I would actually try to normalize the tables first into webservices, webservice_tags, tags, and blog_posts, like this:
webservices:
-------------------------
| id | webservice_name |
=========================
| 1 | Test Service |
-------------------------
| 2 | Example Service |
-------------------------
| 3 | Another Service |
-------------------------
webservice_tags:
------------------------------------------
| id | tag_name | webservice_id |
==========================================
| 1 | TEST | 1 |
------------------------------------------
| 2 | TESSST | 1 |
------------------------------------------
| 3 | TES | 1 |
------------------------------------------
| 4 | OPTION | 2 |
------------------------------------------
| 5 | OPT | 2 |
------------------------------------------
| 6 | EXAMPLE | 2 |
------------------------------------------
| 7 | ANOTHER | 3 |
------------------------------------------
| 8 | ANO | 3 |
------------------------------------------
| 9 | THER | 3 |
------------------------------------------
| 10 | FOO | NULL |
------------------------------------------
| 11 | BAR | NULL |
------------------------------------------
Note that I'm using tag_name in the pivot table above for the sake of readability. I guess it would be better to use tag_id instead.
tags:
--------------------------
| id | tag_name |
==========================
| 1 | TEST |
--------------------------
| 2 | TESSST |
--------------------------
| 3 | TES |
--------------------------
| 4 | OPTION |
--------------------------
| 5 | OPT |
--------------------------
| 6 | EXAMPLE |
--------------------------
| 7 | ANOTHER |
--------------------------
| 8 | ANO |
--------------------------
| 9 | THER |
--------------------------
| 10 | FOO |
--------------------------
| 11 | BAR |
--------------------------
blog_posts:
-----------------------------------------------------------
| id | title | blog_id | tag_name |
===========================================================
| 1 | Blog Title 1 | 1 | THER |
-----------------------------------------------------------
| 2 | Blog Title 2 | 1 | TES |
-----------------------------------------------------------
| 3 | Blog Title 3 | 1 | ANOTHER |
-----------------------------------------------------------
Now to get a report of how many blog posts are created by each web service, we can do this with a join query.
In this case a QueryBuilder query would give a better performance since we want to know just the aggregates, not the actual database models:
$report = DB::table('webservices')
->leftJoin('webservice_tags', 'webservice_tags.webservice_id', '=', 'webservices.id')
->leftJoin('tags', 'tags.tag_name', '=', 'webservices_tags.tag_name')
->leftJoin('blog_posts', 'blog_posts.tag_name', '=', 'tags.tag_name')
->where('blog_posts.blog_id', '=', $blog->id)
->select(['webservices.webservice_name', DB::raw('COUNT(`blog_posts.id`) as `num_posts`')])
->groupBy('webservices.id')
->get();
You would now get a report of how many blog posts were created by each web service, for all web services.
One note is that, since you mentioned a web service may share tags, and you are identifying web service <-> blog posts via tags,
you cannot 100% accurately determine the origin web service because you won't know for sure which web service set that tag for that post.
Extra: You might notice that I also added FOO and BAR to the Webservice Tags.
This would help in case, like you said that web services are "very inconsistent", new tags would be added to your database.
You could also query them and generate a report easily to find out which tag hasn't been mapped yet.