I'm fairly new to Tridion and I have to implement functionality that will allow a content editor to create a component and assign multiple date ranges (available dates) to it. These will need to be queried from the broker to provide a search functionality.
Originally, this was only require a single start and end date and so were implemented as individual meta data fields.
I am proposing to use an embedded schema within the schema's 'available dates' metadata field to allow multiple start and end dates to be assigned.
However, as the field is now allowing multiple values, the data is stored in the broker as comma separated values in the 'KEY_STRING_VALUE' column rather than as a date value in the 'KEY_DATE_VALUE' column as it was when it was only allowed a single start and end values.
eg.
KEY_NAME | KEY_STRING_VALUE
end_date | 2012-04-30T13:41:00, 2012-06-30T13:41:00
start_date | 2012-04-21T13:41:00, 2012-06-01T13:41:00
This is now causing issues with my broker querying as I can no longer use simple query logic to retrieve the items I require for the search based on the dates.
Before I start to write C# logic to parse these comma separated dates and search based on those, I was wondering if anyone had had similar requirements/experiences in the past and had implemented this in a different way to reduce the amount of code parsing required and to use the broker querying to complete the search.
I'm developing this on Tridion 2009 but using the 5.3 Broker (for legacy reasons) so the query currently looks like this (for the single start/end dates):
query.SetCustomMetaQuery((KEY_NAME='end_date' AND KEY_DATE_VALUE>'" + startDateStr + "') AND (ITEM_ID IN(SELECT ITEM_ID FROM CUSTOM_META WHERE KEY_NAME='start_date' AND KEY_DATE_VALUE<'" + endDateStr + "')))";
Any help is greatly appreciated.
Just wanted to come back and give some details on how I finally approached this should anyone else face the same scenario.
I proposed the set number of fields to the client (as suggested by Miguel) but the client wasn't happy with that level of restriction.
Therefore, I ended up implementing the embeddable schema containing the start and end dates which gave most flexibility. However, limitations in the Broker API meant that I had to access the Broker DB directly - not ideal, but the client has agreed to the approach to get the functionality required. Obviously this would need to be revisited should any upgrades be made in the future.
All the processing of dates and the available periods were done in C# which means the performance of the solution is actually pretty good.
One thing that I did discover that caused some issues was that if you have multiple values for the field using the embedded schema (ie in this case, multiple start and end dates) then the meta data is stored in the KEY_STRING_VALUE column in the CUSTOM_META table. However, if you only have a single value in the field (i.e. one start and end date) then these are stored as dates in the KEY_DATE_VALUE column in the same way as if you'd just used single fields rather than an embeddable schema. It seems a sensible approach for Tridion to take but it serves to make it slightly more complicated when writing the queries and the parsing code!
This is a complex scenario, as you will have to go throughout all the DCPs and parse those strings to determine if match the search criteria
There is a way you could convert that metadata (comma separated) in single values in the broker, but the name of the fields need to be different Range1, Range2, ...., RangeN
You can do that with a deployer extension where you change the XML Structure of the package and convert each those strings in different values (1,2, .., n).
This extension can take some time if you are not familiar with deployer extensions and doesn't solve 100% your scenario.
The problem of this is that you still have to apply several conditions for retrieve those values and there is always a limit you have to set (Versus the User that can add as may values as wants)
Sample:
query.SetCustomMetaQuery((KEY_NAME='end_date1'
query.SetCustomMetaQuery((KEY_NAME='end_date2'
query.SetCustomMetaQuery((KEY_NAME='end_date3'
query.SetCustomMetaQuery((KEY_NAME='end_date4'
Probably the fastest and easiest way to achieve that is instead to use an multi-value field, use different fields. I understand that is not the most generic scenario and there are Business Requirements implications but can simplify the development.
My previous comments are in the context of use only the Broker API, but you can take advantage of a search engine if is part of your architecture.
You can index the Broker Database and massage the data.
Using the Search Engine API you can extract the ids of the Components/Component Templates and use the Broker API to retrieve the proper information
Related
I am in the process of writing an advanced search function using Spring boot and MySQL for a Book Management system.
My Book object contains various information such as material id,book name, author, publisher, description, product type (as in a story book or a reference material etc.)
I managed to write an ExampleMatcher as follows;
ExampleMatcher exampleMatcher = ExampleMatcher.matchingAny().
withIgnoreCase()
.withIgnorePaths("material_id")
.withStringMatcher(ExampleMatcher.StringMatcher.CONTAINING)
.withStringMatcher(ExampleMatcher.StringMatcher.STARTING)
.withIgnoreNullValues();
Example example = Example.of(book, exampleMatcher);
List<Book> all = bookRepository.findAll(example);
But when i get the results set, the results are sorted according to the material id. And records that have attributes matching almost all the fields are also there, but sorted according to the id.
Is there a way for me to sort the results in a way that the most matching records are in the first few records in the list and then the other records? As in, to sort from most matching to least matching?
As far as i understood, JpaSort allows ascending and descending sorting and also we can allow specific sorting for specific attributes.
But in the advanced search, the searching is done dynamically according to the attributes that the user fills in. Therefore, i cannot program which fields of the table to sort right? For example, if i program the book name field to be sorted in ascending order and if the user did not specify any value for that particular field, then sorting under that field is useless right?
That is why i want to know if there is any way to dynamically sort the results from most matching to least matching. Any way of achieving this task is much appreciated. Thank you.
After two whole days of reading more than 50-70 articles and posts on the Internet, i was able to implement the Advanced Search in a more optimized manner.
I was not able to find how to sort the results obtained from most-matching to least-matching as i originally asked in the question. So if someone can still answer my original question, i am happy to accept.
The workaround i used is as follows.
From an idea i got to dynamically generate the SQL query, i was able to find a lead and referred articles on that.
In Dynamic Query in Spring Boot, the author has used Java Reflection API to manually go through the non-null fields of the entity class and to generate the SQL query. But as we all know, when you are using Springboot and when all the configurations are done for you by Springboot, i don't think it is really an effective way to have the Hibernate dependency explicitly, to manage sessions and run your SQL query. The HibernateJpaSessionFactoryBean used in the above article is now deprecated. Therefore, i referred various articles and the Spring Data Jpa Documentation but could not resolve the error that i always got saying that Springboot cannot find the entityManagerFactory bean.
Therefore, i searched for ways to dynamically generate queries using Spring Data JPA itself and not use Hibernate and facing a hassle on session managing etc. Dynamic Queries with Spring Data JPA Specifications and Using Spring Data JPA Specification has enough information on how to implement JpaSpecification in order to generate queries dynamically in Springboot.
So at the end, i used information from all these 3 articles sited here to come up with my implementation. I used Java Reflection to create the Specification according to the Class type of the non-null fields in my entity object.
The new part i added by myself was, i grouped all the separate Specifications together to a List, and wrote a loop to dynamically generate the final Specification to be used in retrieving the data. It is as follows.
List<BookSpecification> bookSpecifications = createDynamicQuery(book);
if (bookSpecifications.size() != 0) {
Specification<Book> dynamicQuery = Specification.where(bookSpecifications.get(0));
for (int i = 1; i < bookSpecifications.size(); i++) {
dynamicQuery = dynamicQuery.or(bookSpecifications.get(i));
}
List<Book> all = bookRepository.findAll(dynamicQuery);
all.forEach(System.out::println);
return all;
}
The createDynamicQuery() method above, which i used in my own way is inspired from the information in the cited articles.
Using this way, i was able to obtain much more accurate Advanced Search results rather than using ExampleMatcher for the same advanced search criteria. And since i am searching by specific field names, the search results were also sorted in an accurate way.
How to generate MySQL Querys with LUIS and fetch data from the DB hosted in Azure?
Should generate a natural language query to an MySQL Query.
e.g.
How much beer was drunken on the oktoberfest 2018?
--> GET amountOfBeer FROM Oktoberfest WHERE Year ==2018;
Does anyone has an idea how to get this to work?
Already generated small Intents in LUIS e.g. GetAmountOfBeer
Dont know how to generate the MySQL Statements and how to get the data from the DB.
Thanks.
You should be able to achieve this, or something similar, using intents and entities. How successful this can be depends on how many and how diverse your queries need to be. First lets start with the phrase you mentioned: "How much beer was drunken on the oktoberfest 2018". You can easily (as you've done) add this as an utterance for an intent, GetAmountOfBeer. Though I'm a fan of intent names that you can read as "I want to GetAmountOfBeer", here you may want to name the intent amountOfBeer so you can use it in your query directly.
Next you need to set up you entities. For year (or datetime rather) that should be easy, as I believe there are some predefined entities for this. I think you need to use a datetime recognizer to parse out the right attribute (like year), but I haven't tried to do this before. Next, Oktoberfest seems to be a specific holiday or event in your DB, so you could create a list entity of all the events you have.
What you are left with is something like (pseudocode) GET topIntent FROM eventEntity WHERE Year ==datetime.Year, or something like that.
If your query set is more complex, you might have to have multiple GET statements, but you could put those in a switch statement by topIntent so that, no matter what the intent is, you can parse out the correct values. You also might want to build this into a dialog where you can check if the entities exist, and if not, you can prompt the user for the missing data.
I need to use Solr for a very quick demo, I have a MySql database that contains 37k records of products online (like gmail, google analytic) where I have information like name, description, and keywords.
I managed to store the data like this structure
{
"keywords":"[\"music-streaming,streaming,internet-radio,audio-scrobbling\"]",
"description":"Last.fm is a music community website that offers personalized internet radio, using a recommendation system called \"Audioscrobbler\" to build a detailed profile of users based on their music tastes and interests. The service...",
"operatingSystem":"[\"Mac,Windows,Linux,Web/Cloud,Android,iPhone,WindowsPhone,KindleFire\"]",
"meta":"[\"Freemium\", \"Mac\", \"Windows\", \"Linux\", \"Web/Cloud\", \"Android\", \"iPhone\", \"...\", \"WindowsPhone\", \"KindleFire\"]",
"name":"Last.fm",
"id":39145,
"category":"audio-and-music"}
Meta & operating system are JSON arrays, while the remaining fields are text fields.
I need help in three things
Is this data structure (schema) is good in terms of structure, searching, and indexing?
I want to build a query where is shows relevant products based on keywords?
How can I turn the fields meta and operating system into filters rather than search keywords?
My final goal is to have a search bar where a user can type in a specific keyword then filter according to operating system and meta
The fields with multiple values should probably be indexed as separate terms in a multiValued field, so that you can query / filter for fields with a specific value. i.e. index the field as 'Mac', 'Windows', 'Linux', 'Web/Cloud', etc., and not as a single value with all the values embedded.
Depending on your exact requirements, similar / relevant documents can be found using the MoreLikeThis component.
When the fields are properly multivalued (as they should be) you can generate a Facet on the field for filtering (and then use fq to filter the result set accordingly).
I have a resource that has a availability field that lists what hours of a day its available for use?
eg. res1 available between 0-8,19-23 hours on a day, the range here can be comma separated values of hour ranges. e.g are 0-23 for 24 hour access, 0-5,19-23 or 0-5,12-15,19-23
What's the best way to store this one? Is char a good option? When the resource is being accessed, my php needs to check the current hour with the hour defined here and then decide whether to allow this access or not. Can I ask mysql to tell me if the current hour is in the range specified here?
I'd store item availability in a separate table, where for each row I'd have (given your example):
id, startHour, endHour, resourceId
And I'd just use integers for the start and end times. You can then do queries against a join to see availability given a certain hour of the day using HOUR(NOW()) or what have you.
(On the other hand, I would've preferred a non-relational database like MongoDb for this kind of data)
1) create a table for resource availability, normalized.
CREATE TABLE res_avail
{
ra_resource_id int,
ra_start TIME,
ra_end TIME
# add appropriate keys for optimization here
)
2) populate with ($resource_id, '$start_time', '$end_time') for each range in your list (use explode())
3) then, you can query: (for example, PHP)
sql = "SELECT ra_resource_id FROM res_avail where ('$time' BETWEEN ra_start AND ra_end)";
....
I know this is an old question, but since v5.7 MySQL supports storing values in JSON format. This means you can store all ranges in one JSON field. This is great if you want to display opening times in your front-end using JavaScript. But it's not the best solution when you want to show all places that are currently open, because querying on a JSON field means a full table scan. But it would be okay if you only need to check on for one place at the time. For example, you load a page showing the details of one place and display whether it's open or closed.
I'm working with a third party software package that is on it's own database. We are using it for the user management back bone on our application. We have an API to retrieve data and access info.
Due to the nature of information changing daily, we can only use the user_id as a pseudo FK in our application, not storing info like their username or name. The user information can change (like person name...don't ask).
What I need to do is sort and filter (paging results) one of my queries by the person's name, not the user_id we have. I'm able to get an array of the user info before hand. Would my best bet be creating a temporary table that adds an additional field, and then sorts by that?
Using MySQL for the database.
You could adapt the stored procedure on this page here to suit your needs the stored procedure is a multi purpose one and is very dynamic, but you could alter it to suit your needs for filtering the person table.
http://weblogs.asp.net/pwilson/archive/2003/10/10/31456.aspx
You could combine the data into an array of objects, then sort the array.
Yes, but you should consider specifically where you will make the temporary table. If you do it in your web application then your web server is stuck allocating memory for your entire table, which may be horrible for performance. On the other hand, it may be easier to just load all your objects and sort them as suggested by eschneider.
If you have the user_id as a parameter, you can create a user defined function which retrieves the username for you within the stored procedure.
Database is on different servers. For all purposes, we access it via an API and the data is then turned into an array.
For now, I've implemented the solution using LINQ to filter and out the array of objects.
Thanks for the tips and helping me go in the right direction.