I'm using Cypress to run a suite of automated tests.
The current version of cypress provides mocha-junit-reporter out of the box, and provides configuration options to pass to the reporter. One of the options is the 'mochaFile'.
I'm using the recommended [hash] tag to output reports across multiple spec files.
this results in a flat mess of files that look like 'results/test-output-abc12345.xml'.
What I want instead is for the test file's relative path and filename to be pass in as the reporter's output file path.
This would give me a structured, feature first view of the output, and in Azure Dev Ops, which aggregates the test output, it would give me correct filenames to detect intermittently failing tests.
Things I've tried that haven't worked:
I've tried to use hooks to modify Cypress's config or set environment variables to try to override the reporterOptions/mochaFile per test at (hopefully) the right time.
I've tried to grab the outputted defaultly-named xml file, and copy it to the correct path+filename given the Cypress.spec.name context, but I can't seem to find the right hook or time to do this.
after and afterEach don't work - I don't think the test report has saved the file yet.
Using a plugin, hooking to some event on test:before:run or test:after:run seem promising, but I'm flying blind since I can't debug into it, so I've been unsucessful in modifying the reporter's output path or copying the file.
I'd love it if someone could show a working example using mocha-junit-reporter, or even a different mocha compatible reporter, if the reporter would play well with Azure Dev Ops, and can help me discover intermittently failing tests.
Related
I would like to execute my load tests on the regular basis and make it part of my CI process.
I'd like to run a Visual Studio loadtest from the console and provide the results of the test in human readable format (preferably HTML page).
For executing the test I use MSTest.exe util which produces .trx file to specified location.
e.g.
mstest /TestContainer:LoadTest.loadtest /resultsfile:c:\Temp\MyResults.trx
My problem is to convert this .trx file into something which can be displayed in browser.
I tried several tools mentioned in another topics like trx2html converters and so on. All of them were built for conversion of unit tests and fail to convert load test results correctly.
Any suggestion please?
An alternative would be to use the "archive artifacts" option in Jenkins for the **/*.trx file(s). That way you can click on the link and the trx file will open in Visual Studio.
Of course that means this is only of use to people with Visual Studio installed.
I have been asked to update a system where header information gets injected into a tif via a 3rd party console application. I don't need to worry about that bit.
The part I have been asked to look at it the merge process that generates the header information.
The current file generated by the process is assumed as correct, before I make any changes, so I want to add this as an approved result, from that I can then check that the changes I make will alter the file as expected.
I thought this would be a good opportunity to look at using ApprovalTests
The problem I have is that for what ever reason the links to the videos are considered corruptible (Possibly show me kittens jumping into boxes or something, which will stop me working, which ironically means I slow down my work done because I cannot see any help videos).
What I have been looking at is the Approvals.Verify and Approvals.VerifyFile extensions.
But what appears to be happening is confusing me.
using VerifyFile creates a received file, but the contents of the file are just a line the name of the file I have asked it to verify.
using Verify(new FileInfo("FileNameHere")) does not appear to generate the received file that I need to flag as approved, but the test does return saying that it cannot find the approved tif file.
I am probably using VerifyFile completely wrong and might be looking at using Verify wrong as well.
useful info?
Might be useful to know, that as this is a legacy application, running as a windows service, I have wrapped the service in a harness that allows me to call the routines, so the files are physically being written elsewhere on the machine outside of my control (well there is a config, but the return of the service I call generates a file in a fixed location if it is successful). I have tried copying that into the Unit Test project, but that doesn't appear to help.
Verify(File) and VerifyFile(string) are both meant to verify an existing file. As such they merely setting the received file to the file you pass in. You will still need to move/approval/create the approved file.
Here is the pseudo code and process.
[UseReporter(typeof(DiffReporter), typeof(ClipboardReporter)]
public void TestTiff()
{
string tif = YourProcessToCreateTifFile();
Approvals.VerifyFile(tif);
}
[Note: if you don't have an image diff installed, like TortoiseDiff, you might want to use the FileLauncherReporter]
Run this, once you get the result, move the file over by pasting your clipboard into a cmd window.
It will move the temporary tif to your test directory with the name ClassName.TestTiff.approved.tif
After that the test should pass until something changes.
Happy Testing!
I'm trying to make my tasks run tests in a certain directory. I was looking at sourceSets, however I inferred that they are useful if you are running outside the test/groovy folder. All of my tests are within the test/groovy folder.
I've got a set of Geb tests as well as a set of service tests. I would like to run them both together and independently. Essentially my tree structure would look like this, being able to run all tests.
Test
--gebTest
----firefoxTest
----chromeTest
----ieTest
--servicesTest
----service1Test
----service2Test
----service3Test
----etc.
My file structure is as follows:
project
-src
--test
---groovy
----com
-----acme
------functional <---where my geb tests sit
------services <---umbrella for services
-------service1 <---each unique service
-------service2
-------service3
-------etc
Can anyone lend me a hand. For the life of me I don't know how Gradle picks what tests to execute.
Thank you in advanced.
SourceSets are indeed a solution to your problem, but I notice you only differentiate your tests by their package names. I'm not sure but that may prove problematic with source sets.
Personally I would prefer a directory structure like this anyway
src
-test
--groovy
---functional
----com etc
---services
----com etc
However, if you are attached to your current structure then take a look at Gradle's test filtering support, which will allow you to filter by package name.
http://www.gradle.org/docs/current/userguide/java_plugin.html#sec:java_test
I was trying to add some automated Unit Tests to my project with Bamboo and have been facing some problems. The Unit Tests themselves are done with googletest, which creates an XML file which should be compatible with the JUnit parser.
However, I'm getting the following error when executing Bamboo's JUnit Parser:
02-Apr-2013 12:11:22 Starting task ''Parse UnitTest output' of type 'com.atlassian.bamboo.plugins.testresultparser:task.testresultparser.junit'
02-Apr-2013 12:11:22 Parsing test results...
02-Apr-2013 12:11:22 Failing task since test cases were expected but none were found.
02-Apr-2013 12:11:22 Finished task 'Parse UnitTest output'
This doesn't seem to have anything to do with the .xml file itself, as I've tried a few. This included my own output.xml, generated by googletest and the sample outputs from https://confluence.atlassian.com/display/BAMBOO/JUnit+parsing+in+Bamboo.
I also adapted said files against the two proposed .xsd files, which should match the output that the JUnit Parser expects, but all to no effect.
Update:
Up until now I told the JUnit Parser to look for ${bamboo.build.working.directory}/output.xml
When I tried **/*.xml it worked.
As I understand it now, after very carefully reading the task description, I have to give it a folder. But I can also give it the files, if I do it in ant-style (with a glob?). This is at the very least very confusing and still doesn't fully answer the inital question. So if anyone could enlighten me, please do.
This is a super-old question, I figured I'd add an answer for posterity. As a few people have commented, the configuration value for test output files requires a relative path. The question is, relative to what?
I think the answer to that depends on how you have your source repositories configured, but in general it will be relative to the root of your project. If all else fails, look at where bamboo is putting your source code when it gets checked out; that'll be the directory to which bamboo appends the test output search path.
For the configuration syntax, you're correct that ant-style patterns can be used (Learning Ant path style for reference).
Just as an example, if you have a project which on your local machine lives at C:\git\MyProject, and your test results end up at C:\git\MyProject\Output\Tests\output.xml, then you'd specify Output/Tests/output.xml in the 'Specify custom results directories' field of the appropriate task configuration. You could also use Output/**/*.xml to search for all .xml files in the Output directory.
I have a couple of test files written in my DSL in my tests plugin/project. Most of the tests use inline multi-line strings and Xtend but in four cases, I need to test code which does some magic with URLs and the classpath, so I really need resources in the classpath for that.
Since loading the resources only works when the extension is correct, I can't give the files a fake extension.
Now my problem: My DSL also has a code generator. This means that eventually, I end up with a couple of generated files in places where I can't have them (they don't compile, for example, and one even contains an error to test error handling when information is split across several files).
I can't disable the Xtext nature because the tests project uses Xtend so for these files, I do need code generation.
Since the generator runs inside Eclipse (I have the DSL plugins installed for other projects), there is no way to override the code generator in Guice.
How can I disable the code generator in this case?
There is a simple way to achieve this:
Open the properties of your project
Expand the entry for your DSL
Select "Compiler"
Select "Enable project specific settings"
Disable/deselect "Compiler is activated" under "General"
If you don't have a properties entry for your DSL:
Add this fragment to your .mwe2 workflow file:
fragment = generator.GeneratorFragment {}
Regenerate your projects
Merge the new code from plugin.xml_gen into plugin.xml both in the base and the UI plugins.
The interesting parts are the two extension points org.eclipse.ui.preferencePages and org.eclipse.ui.propertyPages