I have been using Test Cafe to write an internal test framework where actions (t.click) and assertions (t.expect) are not directly written inside the spec, but are defined and aggregated in other files.
Everything cool until a test does not fail: in this case the Test Cafe reporter writes in console the assertion/action failed and the relative snippet of code, but I did not find the way to understand the full stack trace of function calls from my test down to the failed assertions.
How can I make sure to provide a full stack trace in the reporter, logging a stack trace with all the calls to function that made my test fail?
I understood that the reason should be linked to how async/await is transpiled into generators: the stack trace of the error shows only the last await executed and not all the previous calls.
<section> ... </section>
<section class="section--modifier">
<h1> ... </h1>
<div>
...
<button class="section__button">
<div class="button__label">
<span class="label__text">Hello!</span> <-- Target of my test
</div>
</button>
...
</div>
</section>
<section> ... </section>
//
// My spec file
//
import { Selector } from 'testcafe';
import {
verifyButtonColor
} from './button';
fixture`My Fixture`
.page`....`;
test('Test my section', async (t) => {
const MySection = Selector('.section--modifier');
const MyButton1 = MySection.find('.section__button');
const MySection2 = Selector('.section--modifier2');
const MyButton2 = MySection2.find('.section__button');
....
await verifyButtonColor(t, MyButton1, 'green'); // it will fail!
....
....
....
await verifyButtonColor(t, MyButton2, 'green');
});
//
// Definition of assertion verifyButtonColor (button.js)
//
import { Selector } from 'testcafe';
import {
verifyLabelColor
} from './label';
export async function verifyButtonColor(t, node, expectedColor) {
const MyLabel = node.find('.button__label');
await verifyLabelColor(t, MyLabel, expectedColor);
}
//
// Definition of assertion verifyLabelColor (label.js)
//
export async function verifyLabelColor(t, node, expectedColor) {
const MyText= node.find('.label__text');
const color = await MyText.getStyleProperty('color');
await t.expect(color).eql(expectedColor, `Color should be ${expectedColor}, found ${color}`); // <-- it will FAIL!
}
What I get not in the reporter is that my test failed because the assertion defined in "verifyLabelColor" failed (the color is not green :(),
...
await t.expect(color).eql(expectedColor, `Color should be ${expectedColor}, found ${color}`);
...
but in the reporter I have no evidence that failed due to the following stack of calls
- await verifyButtonColor(t, MyButton1, 'green');
- await verifyLabelColor(t, MyLabel, expectedColor);
- await t.expect(color).eql(expectedColor, `Color should be ${expectedColor}, found ${color}`);
Any body faced a similar problem?
An alternative could be to log the "path" of the selector that caused the failure, but looking to Test Cafe documentation I did not find the possibility to do it: knowing that the assertion failed on element with the path below could at least help to understand what went wrong
.section--modifier .section__button .button__label .label__text
This subject is related to TestCafe proposal : Have a multiple stacktrace reporter for fast analysis when a test fails
In the meantime you could give a try to this reporter: /testcafe-reporter-cucumber-json or maybe you could develop your own reporter
Related
We have a web component under #shadow-root that i want to test using testcafe .
But when i try to load the page from the testcafe script, page loads but only #shadow-root component is not loading and any assertion to verify the existence of those elements fails.
Console error shows " hammerhead.js: Uncaught DOMException: Failed to execute 'setAttribute' on 'Element': 'hammerhead|element-processed' is not a valid attribute name."
Anyone knows how to enable the loading of #shadow-root components on the webpage using testcafe ?
It works fine with this example:
import { Selector } from 'testcafe';
fixture`Selector.shadowRoot`
.page`https://devexpress.github.io/testcafe/example/`;
test('Get text within shadow tree', async t => {
const shadowRoot = Selector('div').withAttribute('id', 'shadow-host').shadowRoot();
const paragraph = shadowRoot.child('p');
await t.expect(paragraph.textContent).eql('This paragraph is in the shadow tree');
await t.eval(() => {
paragraph().style.display = 'block';
}, {
dependencies: { paragraph }
})
await t.click(paragraph);
});
Refer to the following thread to see how to work with shadowRoot in TestCafe: https://testcafe.io/documentation/402829/guides/basic-guides/element-selectors?search#access-the-shadow-dom.
If this doesn't help, please share a simple sample based on the following template: https://github.com/DevExpress/testcafe/issues/new?assignees=&labels=TYPE%3A+bug&template=bug_report.yaml.
I am currently working on a GitLab CI test environment and I have a test harness which we use to test our SDK. I have gone about setting up a custom event that is fired on the page which designates the end of the test run. In my puppeteer implementation I am wanting to listen for this custom event "TEST_COMPLETE".
I have not been successful in getting this to work so I figured I would at least make sure the custom-event.js example on the puppeteer repo worked and there too I am not seeing what I believe I should be getting. I cloned the main repo below and performed an npm install. When I execute the js test below, setting headless:false and don't close the browser, I do not see any log in console that shows any custom event being fired.
It is my understanding that I should see some console event message with 'fired' and then 'app-ready' event and info, but this is not the case. Even if I interact with the page I don't see anything outside of some 'features_loaded' and 'features_unveil' logs.
https://github.com/puppeteer/puppeteer/blob/main/examples/custom-event.js
Anyone able to get the expected behavior on this code today? Not sure if this worked previously and has broke since or I am just doing something wrong. Any info would be of great help, Thanks!
Not sure if this is what you need, but I can get the message 'TEST_COMPLETE fired.' in Node.js console with this simplified code (puppeteer 8.0.0):
import puppeteer from 'puppeteer';
const browser = await puppeteer.launch();
try {
const [page] = await browser.pages();
await page.goto('https://example.org/');
await page.exposeFunction('onCustomEvent', async (type) => {
console.log(`${type} fired.`);
await browser.close();
});
await page.evaluate(() => {
document.addEventListener('TEST_COMPLETE', (e) => {
window.onCustomEvent('TEST_COMPLETE');
});
document.dispatchEvent(new Event('TEST_COMPLETE'));
});
} catch (err) { console.error(err); }
My problem
ava logging (t.log) only work inside a test, but not during setup (before, beforeEach) or teardown (after*) functions.
This means that meaningful setup / teardown data, which is very useful for debugging and reproducing, is lost. This happens both for successful and failed tests, and with and without the --verbose flag.
Code
import test from 'ava';
test.before(t => {
// This runs before all tests
t.log('before - 1');
});
test.before(t => {
// This runs after the above, but before tests
t.log('before - 2');
});
test.after('cleanup', t => {
// This runs after all tests
t.log('after');
});
test.after.always('guaranteed cleanup', t => {
// This will always run, regardless of earlier failures
t.log('after always');
});
test.beforeEach(t => {
// This runs before each test
t.log('beforeEach');
});
test.afterEach(t => {
// This runs after each test
t.log('afterEach');
});
test.afterEach.always(t => {
// This runs after each test and other test hooks, even if they failed
t.log('afterEachAlways');
});
test(t => {
t.log('A test');
t.pass();
});
test(t => {
t.log('A test');
t.fail();
});
Output
$ ava run.js --verbose
✔ [anonymous]
ℹ A test
✖ [anonymous] Test failed via `t.fail()`
ℹ A test
1 test failed [00:22:08]
[anonymous]
ℹ A test
/Users/adam/Personal/tmp/ava-bug-log-in-before-each/run.js:46
45: t.log('A test');
46: t.fail();
47: });
Test failed via `t.fail()`
Note that only the printouts from the test (A test) are show. All other logs are lost.
My question
How can I see the logs from the setup and teardown steps of the test suite?
Could you open an issue for this? https://github.com/avajs/ava/issues
I agree this should work.
I'm writing a unit test of an AngularJS 1.x directive.
If I use "template" it works.
If I use "templateUrl" it does not work (the directive element remains the same original HTML instead of being "compiled").
This is how I create the directive element to test in Jasmine:
function createDirectiveElement() {
scope = $rootScope.$new();
var elementHtml = '<my-directive>my directive</my-directive>';
var element = $compile(elementHtml)(scope);
scope.$digest();
if (element[0].tagName == "my-directive".toUpperCase()) throw Error("Directive is not compiled");
return element;
};
(this does not actually work, see Update for real code)
I'm using this workaround to use the $httpBackend from ngMockE2E (instead of the one in ngMock). In the browser developer "network" tab I don't see any request to the template file. It seems to work because I solved the error "Object # has no method 'passThrough'".
I know that the call to the template is done asynchronously using the $httpBackend (this means $compile exit before the template is really applied).
My question is:
obviously $compile is not doing what I expect. How can I trap this error?
If I use a wrong address in the templateUrl I don't receive any error.
How can I found the problem happened when I called $compile(directive) or scope.$digest() ?
Thanks,
Alex
[Solution]
As suggested by #Corvusoft I inject $exceptionHandler and I check for errors after every test.
In the end this is the only code I have added:
afterEach(inject(function ($exceptionHandler) {
if ($exceptionHandler.errors.length > 0)
throw $exceptionHandler.errors;
}));
Now I can clearly see the errors occurred in the Jasmine test result (instead of search for them in the console), example:
Error: Unexpected request: GET /api/category/list
No more request expected,Error: Unexpected request: GET /api/category/list
No more request expected thrown
And, most important, my tests does not pass in case there are some errors.
[Update to show real example case]
Actually the real code to make templateUrl work use asynchronous beforeEach ("done") and a timeout to wait the end of compile/digest.
My directive use some prividers/services and the template contains other directives which in turn use their templateUrl and make calls to some APIs in the link function().
This is the current (working) test code:
// workaround to have .passThrough() in $httpBackend
beforeEach(angular.mock.http.init); // set $httpBackend to use the ngMockE2E to have the .passThrough()
afterEach(angular.mock.http.reset); // restore the $httpBackend to use ngMock
beforeEach(inject(function (_$compile_, _$rootScope_, _$http_, $httpBackend, $templateCache, $injector) {
$compile = _$compile_;
$rootScope = _$rootScope_;
$http = _$http_;
$httpBackend.whenGET(/\/Scripts of my app\/Angular\/*/).passThrough();
$httpBackend.whenGET(/\/api\/*/).passThrough(); // comment out this to see the errors in Jasmine
}));
afterEach(inject(function ($exceptionHandler) {
if ($exceptionHandler.errors.length > 0)
throw $exceptionHandler.errors;
}));
beforeEach(function(done) {
createDirectiveElementAsync(function (_element_) {
element = _element_;
scope = element.isolateScope();
done();
});
});
function createDirectiveElementAsync(callback) {
var scope = $rootScope.$new();
var elementHtml = '<my-directive>directive</my-directive>';
var element = $compile(elementHtml)(scope);
scope.$digest();
// I haven't found an "event" to know when the compile/digest end
setTimeout(function () {
if (element.tagName == "my-directive".toUpperCase()) throw Error("Directive is not compiled");
callback(element);
}, 0.05*1000); // HACK: change it accordingly to your system/code
};
it("is compiled", function () {
expect(element).toBeDefined();
expect(element.tagName).not.toEqual("my-directive".toUpperCase());
});
I hope this example helps someone else.
$exceptionHandler
Any uncaught exception in AngularJS expressions is delegated to this
service. The default implementation simply delegates to $log.error
which logs it into the browser console.
In unit tests, if angular-mocks.js is loaded, this service is overridden by mock $exceptionHandler which aids in testing.
angular.
module('exceptionOverwrite', []).
factory('$exceptionHandler', ['$log', 'logErrorsToBackend', function($log, logErrorsToBackend) {
return function myExceptionHandler(exception, cause) {
logErrorsToBackend(exception, cause);
$log.warn(exception, cause);
};
}]);
I am creating an element (a router - but that is not important), that is scanning the DOM soon after it has attached for particular other custom elements. I certain cases it needs to throw an error and I want to test for these.
The test I constructed is not failing - but as far as I can make out the test has already failed before my element gets attached. I suspect it is the asynchronous nature of things.
Here is the snippet of the test in question. The test fixture in question contains elements that will cause one of the elements to fail after a 'dom-change' event happens (which it has a listener for) when it then scans the dom for other things.
it('should fail if two route elements both designate thenselves as home', function(done) {
var t= document.getElementById('multiple_home');
function multiple () {
t.create();
}
expect(multiple).to.throw(Error);
t.restore();
done();
});
I think the problem is related to the fact that the fixture is created in multiple, but hasn't yet failed by the time multiple exits. I am wondering if I can pass a Promise to expect - except I am not sure how to turn mulitple into a Promise to try it out.
I eventually found a way, but it requires instrumenting the element a bit to support this.
In the elements "created" callback I create a Promise and store the two functions to resolve and reject it in "this" variables - thus:-
this.statusPromise = new Promise(function(resolve,reject){
this.statusResolver = resolve;
this.statusRejector = reject;
}.bind(this));
In the DOM parsing section I use a try catch block like this
try {
//parse the dom throwing errors if anything bad happens
this.statusResolver('Any useful value I like');
} catch (error) {
this.statusRejector(error);
}
I then made a function that returns the promise
domOK: function() {
return this.statusPromise;
}
Finally in my test I was now able to test something like this (I load the fixture in each test, rather than a beforeEach, because I am using a different fixture for each test. I do clear it down again in an afterEach). Note the use of the .then and .catch functions from the Promise.
it('should fail if two route elements declare the same path name',function(done){
t = document.getElementById('multiple_path');
t.create();
r = document.getElementById('router')
r.domOK().then(function(status){
//We should not get here throw an error
assert.fail('Did not error - status is: ' + status);
done();
}).catch(function(error){
expect(error.message).to.equal('There are two nodes with the same name: /user');
done();
});