How to test SQL queries/reports? - mysql

Question

We are developing a Rails app, that has quite a few pages with data reports. A typical reporting page is based on a relatively big SQL query, usually involving 5–8 table joins.
The cornerstone question we've stumbled upon is – writing integration tests reports pages. A common integration test of ours looks like this:
creating a bunch of records in the DB via factory_girl in the test setup,
fire up a capybara scenario, where a user logs in, advances to the page with report, and sees the right data in it.
As the app grows and we get to create more of such reports pages, we've started to run into the following problem - the setup for each individual test ends up being too big, complex and generally hard to read and maintain.
Creating such a test significantly raises the bar for a developer in delivering a feature, related to reporting, as it is very time-consuming and not optimized for happiness. However, we still need to make sure our reports are correct.
Therefore, my questions are:
should or should we not test pages with reports?
if we should test the reports, then what would be the least painful way to do that?
where are we doing wrong?

Answer 1

1. should or should we not test reports page?
You should definitely test your reports page.
2. if we should test the reports, then what would be the least painful way to do that?
Given the size of the reports, you're probably going to have the following problems:
your tests will become super slow;
your tests will become really hard to read and maintain due to the huge setup;
your tests will stop being updated when reports change.
With this, you'll probably stop maintaining properly your specs.
So, first, you should differentiate between:
testing that UI shows proper results (acceptance) vs
testing that reports are generated correctly (unit and integrated).
The tests for the first scenario, UI, which use Capybara, should test UI and not reports themselves. It'd cover that reports data is showing as they were generated by their respective classes, which make us conclude that you don't need to test the millions of report lines, but rather that the table has the correct columns and headers, pagination is working etc. You'd test that the first, second and maybe last report line are showing properly.
On the other hand, the tests for the second scenario, reports generation, should test that reports are generated. That has nothing to do with the UI, as you could be serving those reports as JSON, HTML, Cap'n Proto and any visualization mean. As an imagination exercise, picture testing reports via JSON responses, then all over again via HTML, then all over again via some other method. It'd become evident that report generation is repeated all over.
This means that report generation is the core and should be tested on its own. Which means you should cover it mainly by unit tests. Tons of them if you need. Huge arrays.
With this setup, you'd have blazingly fast unit tests covering your reports and their edge cases, a few integrated tests making sure report generation pieces are connected properly and a few acceptance tests covering your UI (Capybara).
Remember the Test Pyramid?
3. where are we doing wrong?
I don't have all the details about your setup, but it seems the main misconception is thinking that reports are the pages themselves. Remember that you could generate reports as CSV or XML and they'd still be the same report internally. In software, a report will probably end up being an array with values.
So, next time, think about separating concepts. You have reports generation and you have the UI. Test them separately and then add some tests in between to make sure they're both integrated well.
In the future, say you move to a Single Page JS App™ and iOS app, you'd not have to get rid of your report generation tests, but UI tests would go into the clients. That's proof that UI is different from reports generation.

Answer 2

I will post ideas we've had so far.
Do not do an integration test at all
Instead writing an integration test, write a lower-level functional test, by treating DB interaction as an interaction with a 3rd party API.
This is how it would be:
stub the object that sends a query to DB with a mock of DB result we're expecting,
rely any test that needs the data – on that result mock,
execute a SQL query expecting empty dataset, although verifying that:
the SQL raised no syntax error,
the result object returned correct columns,
the column types are as what we are expecting them to be.
This advantages of this approach are:
the tests setup is no longer a cognitive barrier,
the tests are significantly faster than they were back in the factory_girl epoch,
if something changes in the DB result (column names set or column types), we still catch that by performing a "real" DB request.
Pre-load data fixtures
Instead of setting up complex tree of records for each and every report test, pre-populate a whole "field" of DB records prior to the entire test suite.
This is how it would be:
before a suite (or the part of it which contains tests for reports), populate the DB with a log of test data
test each and every report without any further setup needed.
This advantages of this approach are:
the tests setup is no longer a cognitive barrier – the data is just always there for you,
the tests are much faster than they were back in the factory_girl epoch.
This disadvantage of this approach is:
occasionally, as new reports come up, one will have to go and add more data to the "fixtures" setup, which will very likely to break existing tests. Fixing existing tests may lead to strange / not readable pull request changesets for the new features.