ruby on rails what is vs what was architecture without papertrail - mysql

Question

Concise explanation
There is a row in the database which shows the current state of 'Umbrella', forged from the Model 'Product'.
You want to access the complete history of what you deem to be relevant changes to Umbrella, involving related models, quickly and painlessly.
The problem is that paper trail doesn't bring in the beef when the events table is tens of thousands of rows long, and you can't truncate it as it contains important history, and its performance is woeful as it has to parse thousands of lines of YAML to find 'relevant' changes.
Background reading done, still no idea what the problem is called
This seems like something basic to me but I see no mention of others tackling it beyond using papertrail, thus I don't know what its non-proprietarily commonly referred to as, if at all. "ruby on rails what is vs what was architecture without papertrail" was the best title I could think of. I'm creating a one to many relationship between models and time?
Have read "A!!! Design Patterns in Ruby, 2007" which references gang of four's design patterns, no mention of this problem?
Have tried "paper trail" gem but it doesn't quite solve it
The problem
Assuming you have Products, Companies and Categories, and
Product: id, name, price, barcode, (also company_id and category_id)
Company: id, name, registered_company_number
Category: id, name, some_immutable_field
Company has many Products
Category has many Products
And you need to see history of each Product, including changes on itself such as price, changes to which company it belongs to, changes to company name, same thing for categories, such as:
date | event | company name | cmp | category | cat | name | price
| | | id | name | id | |
------|---------------|--------------|-----|----------|-----|----------|------
jan11 | created | megacorp | 1 | outdoors | 101 | umbrella | 10
feb11 | cat change | megacorp | 1 | fashion | 102 | umbrella | 10
mar11 | cat rename | megacorp | 1 | vogue | 102 | umbrella | 10
apr11 | cmp rename | megacorp inc | 1 | vogue | 102 | umbrella | 10
may11 | cmp change | ultra & sons | 2 | vogue | 102 | umbrella | 12
jul11 | cmp change | megacorp | 1 | vogue | 102 | umbrella | 12
note that whilst umbrella was with ultra & sons, megacorp inc changed its name back to megacorp, but we don't show that in this history as its not relevant to this product. (The name change of company 1 happens in jun11, but is not shown)
This can be accomplished with papertrail, but the code to do it is either very complex, long and procedural; or if written 'elegantly' in the way papertrail intended, very very slow as it makes many db calls to what is currently a very bloated events table.
Why paper trail is not the right solution here
Paper trail stores all changes in YAML, the database table is polymorphic and stores a lot of data from many different model. This table and thus this gem seems to be suited to identify who did what changes... but to use it for history like I need to use it, its like a god table that stores all information about what was and has too much responsibility.
The history I am after does not care about all changes to an object, only certain fields. (But we still need to record all the small changes, just not include them in the history of products, so we can't just not-record these things as paper trail has its regular duties identifying who did what, it cannot be optimised solely for this purpose). Pulling this information requires getting all records where the item_type is Product, where the item_id is of the currently being viewed product_id, then parsing the YAML, and seeing if we are interested in the changes (is a field changed, which is a field we are interested in seeing the changes to?). Then doing the same for every category and company that product has been associated with in its lifetime, but only keeping the changes which occur in the windows for which product has been associated to said category/company.
Paper trail can be turned off quite easily... so if one of your devs were to disable it in the code somewhere as an optimisation whilst some operations were to be run, but forget to write the code to turn it back on, no history recorded. And because paper trail is more of a man on the loop than man in the loop, if its not running you might not notice (then have to write overly complex code which catches all the possible scenarios with holey data). A solution which enforces the saving of history is required.
Half baked solution
Conceptually I think that the models should be split between that which persists and that which changes. I am surprised this is not something baked into rails from the ground up, but then there are some issues with it:
Product: id, barcode
Product_period: id, name, price, product_id, start_date, (also company_id and product_id)
Company: id, registered_company_number
Company_period: id, name, company_id, start_date
Category: id, some_immutable_field
Category_period: id, name, category_id, start_date
Every time the price of the product, or the company_id of the product changes, a new row is added to product_period which records the beginning of a new era where the umbrella now costs $11, along with the start_date (well, time) that this auspicious period begins.
Thus in the product model, all calls to things which are immutable or we only care about what the most recent value is, remain as they are; whereas things which change and we care, have methods which to an outsider user (or existing code) appear to be operating on product model, but in fact make a call to most recent product_period for this product and get the latest values there.
This solves the problem superficially, but its a little long winded, and it still has the problem that you have to poke around through company_period and category_period selecting relevant entries (as in the company/category experiences a change and it is during a time when product was associated with it) rather than something more elegant.
At least the MySQL will run faster and there is more freedom to make indexes, and there is no longer thousands of YAML parses bogging it down.
On the quest to write more readable code, are these improvements sufficient? What do other people do? Does this have a name? Is there a more elegant solution or just a quagmire of trade offs?

Answer 1

There are a bunch of other versioning and history gems for rails (I contributed to the first one, 10 years ago!) - find them here, https://www.ruby-toolbox.com/categories/Active_Record_Versioning
They all have different methods for storing like you suggest above, and some are configurable. I also don't agree with the polymorphic god table for all users, but it's not too slow if you have decent indexes.