Im using phpunit & phpundercontrol to run the RC Selenium on every build.
PHPUnit allows you to implement your own TestListener. Custom test listeners implement the abstract methods in the PHPUnit_Framework_TestListener interface. Specifically, your listener will implement:
startTestSuite()
endTestSuite()
startTest()
endTest()
addError()
addFailure()
addSkippedTest()
addIncompleteTest()
Once you've attached the TestListner these methods will be called each time the corresponding events occur in your test suite. These methods will be written to perform the INSERTs and UPDATEs on a test results database that you'll create.
Attaching the listener class to your suite is as easy as adding a tag to the phpunit.xml configuration file. For example:
<phpunit>
<testsuites>[...]</testsuites>
<selenium>[...]</selenium>
<listeners>
<listener class="Database"
file="/usr/loocal/share/pear/PHPUnit/Util/Log/Database.php">
</listeners>
</phpunit>
That's all you need!
In fact, PHPUnit already comes with a working version of the listener I just described (PHPUnit_Util_Log_Database), as well as two different database schema definitions.
On many systems this class will live at /usr/loocal/share/pear/PHPUnit/Util/Log/Database.php, and the schemas at /usr/loocal/share/pear/PHPUnit/Util/Log/Database/MySQL.sql and /usr/loocal/share/pear/PHPUnit/Util/Log/Database/SQLite3.sql. You may have to do some tweaking depending on the DBMS you're using.
See these sections of the documentation (it wont let me post two links:
http://www.phpunit.de/manual/3.4/en/extending-phpunit.html#extending-phpunit.PHPUnit_Framework_TestListener
htp://www.phpunit.de/manual/3.4/en/api.html#api.testresult.tables.testlistener
(StackOverflow won't let me post two links, so you'll have to correct the HTTP in that second one)
I am working on the same problem.
Have asked a related question here a few days ago.
My attempt using Selenium IDE, Selenium RC and perl.
General strategy:
You can make newer releases of phpunit generate TAP output (options --tap, --log-tap).
(TAP is Test Anything Protocol - standardized output format)
Parse the logfile to obtain the suite metadata from the TAP parser object, insert into database using perl, e.g. "# Number of Passed": , "Failed", "Unexpectedly succeeded",
Related
I have used many discord API wrappers, but as an experienced python developer, unfortunately I somehow still do not understand how a command gets called!
#client.command()
async demo(ctx):
channel = ctx.channel
await channel.send(f'Demonstration')
Above a command has been created (function) and it is placed after its decorator #client.command()
To my understanding, the decorator is in a way, a "check" performed before running the function (demo) but I do not understand how the discord.py library seemingly "calls" the demo function.....?? Is there some form of short/long polling system in the local imported discord.py library which polls the discord API and receives a list of jobs/messages and checks these against the functions the user has created?
I would love to know how this works as I dont understand what "calls" the functions that the user makes, and this would allow me to make my own wrapper for another similar social media platform! Many thanks in advance.
I am trying to work out how functions created by the user are seemingly "called" by the discord.py library. I have worked with the discord.py wrapper and other API wrappers before.
(See source code attached at the bottom of the answer)
The #bot.command() decorator adds a command to the internal lists/mappings of commands stored in the Bot instance.
Whenever a message is received, this runs through Bot.process_commands. It can then look through every command stored to check if the message starts with one of them (prefix is checked beforehand). If it finds a match, then it can invoke it (the underlying callback is stored in the Command instance).
If you've ever overridden an on_message event and your commands stopped working, then this is why: that method is no longer being called, so it no longer tries to look through your commands to find a match.
This uses a dictionary to make it far more efficient - instead of having to iterate over every single command & alias available, it only has to check if the first letters of the message match anything at all.
The commands.Command() decorator used in Cogs works slightly different. This turns your function into a Command instance, and when adding a cog (using Bot.add_cog()) the library checks every attribute to see if any of them are Command instances.
References to source code
GroupMixin.command() (called when you use #client.command()): https://github.com/Rapptz/discord.py/blob/24bdb44d54686448a336ea6d72b1bf8600ef7220/discord/ext/commands/core.py#L1493
As you can see, it calls add_command() internally to add it to the list of commands.
Adding commands (GroupMixin.add_command()): https://github.com/Rapptz/discord.py/blob/24bdb44d54686448a336ea6d72b1bf8600ef7220/discord/ext/commands/core.py#L1315
Bot.process_commands(): https://github.com/Rapptz/discord.py/blob/master/discord/ext/commands/bot.py#L1360
You'll have to follow the chain - most of the processing actually happens in get_context which tries to create a Context instance out of the message: https://github.com/Rapptz/discord.py/blob/24bdb44d54686448a336ea6d72b1bf8600ef7220/discord/ext/commands/bot.py#L1231
commands.Command(): https://github.com/Rapptz/discord.py/blob/master/discord/ext/commands/core.py#L1745
My team is just getting started with X-Ray, and we are setting up our pipelines. However, while doing this I noticed that if I submit a Junit xml file to X-Ray via the REST api, it will create new tests for any test data that isn't already in the system.
Is there a way to have X-Ray ignore test results for tests that don't exist for the test execution? I don't want it constantly creating extra tests.
For example:
(Jira/X-Ray Server) TestExecution MyExecution has test testA
From client, I submit a Junit xml file containing results for testA and testB in the MyExecution TestExecution
testB now exist on the server under MyExecution
I would like to be able to submit the Junit xml file without it creating extra tests.
Whenever you import automation results using the REST API, or any of the available CI plugins, Xray will autoprovision ("generic") Test entities.
The flow is detailed here.
Xray tries to find a unique identifier for the automated test; in the case of JUnit, it's based on the full classname plus the name of the test method; this will become part of the Generic Definition field. The process for JUnit is described in more detail here.
How it works for a different test automation framework/report formats, is similar and is detailed on respective documentation pages.
If a "generic" Test is found, then the Test is reused and a Test Run is created against it. Otherwise, the Test will be auto-provisioned.
This process isn't configurable. However, in theory, if the user that you use for the submission of automation results isn't able to create Test issues, you may have what you need.
Things like these are usually not configurable because they are normally a consequence of applying good practices usually discussed internally with the team(s).
In my Coded UI Test project, I need to check if few Labels or Messages are consistent with the context. But those checks are not critical if not consistent and I need to output them only as warnings.
Note that I'm using nested ordered tests to use only one global ordered test with vstest.console.exe and get in one shot the overall test coverage report.
Till now I was creating assertions to check those consistencies, but an assertion failure leads to Test failure, then to ordered test failure and then to playback stop.
I tried to change Playback.PlaybackSettings.ContinueOnError value before and after the assertion: this works as I expect as the assertion is well reported as a warning in the html report file. But whatever, it causes the ordered test to stop and then my global ordered test chaining to fail...
I tried to use TestContext.WriteLine too instead of creating assert, but it seems that this is not output in the html report.
So my question is:
is there any way to create an assertion only as a Warning that will be output in the html report file and that doesn't lead to a test failure?
Thanks a lot for any answer and help on this ;)
So I got my solution with developping my own Warning Engine to integrate Warnings in test report, 'cause I found no existing solution for that with the current Coded UI Test Assertion engine.
I'll try to take some time to post generic parts of the code structure with comments translated in english (we're french so default comments are french for now...), but here are the main step lines :
Create a template based on the UITestActionLog.html original file
report structure of Coded UI Test engine, with only the start
bloc and the javascript functions and CSS declarations in it.
Create an assertion class with a main function to manage insertion
of Warning html bloc in the html report first created from the template.
Then create custom assert functions to call the main function
whereever on runtime, and custom Stopwatch to inject elapsed time in
the report ('cause I could'nt found a way to get back the elapsed
time directly from the Coded UI Test engine).
That's it.
Just a proposition as a way to do it, maybe not the best one but it worked for me. I'll try to take time to put blocl codes to be clearer on it.
I found that my Sonar instance 5.1 or 5.1.1 (with latest sonar-runner 2.x) stopped showing part of the Unit test info (Unit test widget) on the project's dashboard.
The properties I had were (in Gradle's sonarRunner > sonarProperties section):
property "sonar.junit.reportsPath", "build/test-results/UT"
property "sonar.surefire.reportsPath", "build/test-results/UT"
To fix it, I had to include the following properties as well:
property "sonar.java.junit.reportsPath", "build/test-results/UT"
property "sonar.java.surefire.reportsPath", "build/test-results/UT"
Just FYI: All my Unit tests reports go under build/test-results/UT folder, all Integration Tests result files go unedr build/test-results/IT folder and etc.
I'm wondering if this is due to Gradle version that I'm using (2.3) or is it due to a later version of SonarQube (4.5+) as I have both SQ 5.1 and 5.1.1 instance.
I know SonarQube team started Multi language support since SonarQube version 4.12
Since SonarQube 4.2, it is possible to run an analysis on a multi-language project.
Now, it raises a question. For Getting the same Unit test info for Groovy based projects, do I need to use:
property "sonar.groovy.junit.reportsPath", "build/test-results/UT"
property "sonar.groovy.surefire.reportsPath", "build/test-results/UT"
something like that if my project has Groovy code instead of java?
Searching "**sonar.java.junit.reportPath"** with using double quotes shows No results found in Google and it forces me to try and see google results if I can run the search again without using " double quotes (for this property).
Doing the same in SonarQube site "search box" shows:
No results found for sonar.java.junit.reportPath. Please try one of the following suggestions:
Though in Gradle, inside
sonarRunner task {
.. inside ..
sonarProperties {
... section ... where I define various sonar props..
}
...
}
I can define both sonar.junit.reportPath, sonar.java.junit.reportPath and similarly, sonar.surefire.reporPath and sonar.java.surefire.reportPath and while running sonarRunner task in Gradle, it doesn't error out. Thus it makes me believe that the property variables are valid.
There are also issues with running sonarRunner or stand alone sonar-runner command for a mixed Java and Groovy based project (i.e. source code in Java but tests in Groovy). Setting sonar.language=java,grvy didn't help. I posted this question on stackoverflow but so far I have no perfect result/answer on how to get a full fledged sonar dashboard up and running for a Groovy projects like I get for a Java project.
Groovy project - Sonar - Publish project and Unit + Integration Test code coverage data
PS: I have tried various values for setting sonar.. variables (as far a sonar source, tests, etc, etc properties are concerned, which they have mentioned on their site's docs section)
The only valid property to use as of now is sonar.junit.reportsPath which will tell the java sonarqube plugin where to import your result of unit tests.
For groovy, this is work in progress, see : http://jira.sonarsource.com/browse/SONARGROOV-2
All the other properties you mentioned do not exist and are not taken into account.
I have several Spock test classes grouped together in a package. I am using Junit 4.10. Each test class contains several feature test methods.
I want to perform some setup steps (such as loading data into a DB, starting up a web server) before I run any test case, but only once when the testing starts.
I want this "OneTimeSetup" method to be called only once whether:
I run all the test classes in the package (for example if they are grouped in a Test Suite)
I run a few test classes
I run only one test class
I run only a certain feature method within a test class
From reading other posts on SO, it seems that this is what TestNG's #BeforeSuite does.
I am aware of Spock's setupSpec() and cleanupSpec() methods, but they only work within a given test class. I am looking to do something like "setupTestSuite()." How can this be achieved in Spock?
You can write a global extension, use a JUnit test suite, call a static method in a helper class (say from setupSpec) that does its work just once, or let the build tool do the job.