I've got a build system running Ant Version 1.10.6
The macrodef that I've summarised below is constructed to switch on "run.multi" either it runs a single test method or it does something else (which i've not included as its not relevant).
The issue I have here is that I cannot get it to run just the one method 'test.method' - I've turned the verbose on and can see it all going in and being set correctly, but it runs all the methods regardless!
<echo level="info" message="Single method=#{test.method}" />
<junit printsummary="on" showoutput="true" fork="yes" forkmode="once">
<assertions>
<enable/>
</assertions>
<test unless="${run.multi}"
name="#{test.single}"
methods="#{test.method}"
haltonfailure="no"
haltonerror="no">
<formatter type="plain" usefile="false"/>
</test>
And here is a portion of the output:
[echo] Single method=testSimple
[junit] Running com.test.util.TheTestToRun
[junit] Testsuite: com.test.util.TheTestToRun
[junit] Running on Java 9.0.4
[junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.473 sec
The reason that this was happening is either a bug or a feature in ant - around the fork option on the junit section. set 'fork=no' and the issue goes away:
<junit printsummary="on" showoutput="true" fork="no" forkmode="once">
<test unless="${run.multi}"
name="#{test.single}"
methods="#{test.method}"
haltonfailure="no"
haltonerror="no">
<formatter type="plain" usefile="false"/>
</test>
A portion of the output:
[junit] Running com.test.util.TheTestToRun
[junit] Testsuite: com.test.util.TheTestToRun
[junit] Running on Java 9.0.4
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.12 sec
Related
I have been using GWT 2.4 for one of my projects. Recently, I have upgraded the GWT version to 2.8.2 and the existing Junit is failing for no reason. Please help me troubleshoot the issue.
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.769 sec
------------- Standard Error -----------------
Unknown argument: -standardsMode
Google Web Toolkit 2.8.2
------------- ---------------- ---------------
Testcase: testOne took 0.537 sec
Caused an ERROR Error processing shell arguments
com.google.gwt.junit.JUnitFatalLaunchException: Error processing shell arguments
at com.google.gwt.junit.JUnitShell.getUnitTestShell(JUnitShell.java:696)
at com.google.gwt.junit.JUnitShell.runTest(JUnitShell.java:682)
at com.google.gwt.junit.client.GWTTestCase.runTest(GWTTestCase.java:421)
at com.google.gwt.junit.client.GWTTestCase.run(GWTTestCase.java:247)
[exec] -test.dev:
[exec] [mkdir] Created dir: C:\gitrepos\Python\petroApps\isdBuild\JDK_1.8\isdApps\topaz\gui\impl\web\client\gwt\pres\reports\htmlunit.dev
[exec] [junit] WARNING: multiple versions of ant detected in path for junit
[exec] [junit] jar:file:/C:/gitrepos/Python/petroApps/isdMWare/buildTools/apache/ant/V1.9.6/lib/ant.jar!/org/apache/tools/ant/Project.class
[exec] [junit] and jar:file:/C:/gitrepos/Python/petroApps/isdMWare/java/apis/google/gwt/V2.8.2/lib/gwt-dev.jar!/org/apache/tools/ant/Project.class
[exec] [junit] Running com.verifone.isd.topaz.gui.impl.web.client.gwt.pres.POSPresTestSuite
[exec] [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.769 sec
[exec] BUILD FAILED
In GWT 2.7.0 (RC1) the -standardsMode has been removed.
"Removed flags: -standardsMode, -nostandardsMode, -quirksMode. (GWTTestCase tests are always run in an HTML page in standards mode.)"
See: GWT Release Notes
To solve your problem, remove the argument.
I have created a Maven Project with TestNG for selenium webdriver (Chrome Browser). I run a basic test with TestNG and Chrome browser is successfully opened. Buf if I run the same test with "Maven Test" command and I get an error message.
Could you please help? I have tried many solutions like suggested by Stack Overflow but still unsuccessful :(
I run "Maven Test" for a basic test "Hello World" and the build is successful >>> Evidence that "Maven Test" creates a successful build
I run TestNG test with the command to open Chrome Browser and it works >>> evidence that TestNG works successfully
I run "Maven Test" with the command to open Chrome Browser and it does NOT work >>> getting frustrated with this, tried all solutions appeared in StackOverflow but unsuccesful
I have created testng.xml file
I have used different versions for Selenium-Java and Selenium-Server dependencies as well as for Chromedriver and SureFire plugin but the problem is still there, now I am using the latest for all of them (see POM file below for more details).
I have uninstalled and reinstalled Google Chrome and now I have latest version 73
I have no firewall on my machine
POM file appears below
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>MavenProject</groupId>
<artifactId>MavenProject</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.9.8</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-server</artifactId>
<version>3.141.59</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
</plugins>
</build>
</project>
testng.xml file appears below:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite">
<test thread-count="5" name="Test">
<classes>
<class name="MavenPackage.NewTest"/>
</classes>
</test> <!-- Test -->
My code is below:
public class NewTest {
WebDriver driver;
#Test
public void MyFunction() {
try {
System.setProperty("webdriver.chrome.driver", "C:\\Selenium\\chromedriver\\chromedriver.exe");
driver = new ChromeDriver();
driver.manage().deleteAllCookies();
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
driver.manage().timeouts().pageLoadTimeout(30, TimeUnit.SECONDS);
driver.get("http://www.ebay.com");
Thread.sleep(3000);
driver.quit();
System.out.println("Test passed");
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Error message
Starting ChromeDriver 73.0.3683.68 (47787ec04b6e38e22703e856e101e840b65afe72) on port 49053
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 23.798 s <<< FAILURE! - in TestSuite
[ERROR] MyFunction(MavenPackage.NewTest) Time elapsed: 21.874 s <<< FAILURE!
org.openqa.selenium.WebDriverException:
Timed out waiting for driver server to start.
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: 'DESKTOP-DV883K3', ip: '192.168.1.109', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_201'
Driver info: driver.version: ChromeDriver
at MavenPackage.NewTest.MyFunction(NewTest.java:20)
Caused by: org.openqa.selenium.net.UrlChecker$TimeoutException: Timed out waiting for [http://localhost:49053/status] to be available after 20008 ms
at MavenPackage.NewTest.MyFunction(NewTest.java:20)
Caused by: java.util.concurrent.TimeoutException
at MavenPackage.NewTest.MyFunction(NewTest.java:20)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] NewTest.MyFunction:20 » WebDriver Timed out waiting for driver server to start...
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 28.467 s
[INFO] Finished at: 2019-03-30T23:32:42Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project MavenProject: There are test failures.
Welcome to SO.
org.openqa.selenium.WebDriverException: Timed out waiting for driver server to start.
This indicates that ChromeDriver is unable to initiate new chrome browser session.
Please update your java to latest version (as per message we can see the you are running java.version: '1.8.0_201' which is older,consider updating to latest java version 11.0)
The possible reasons:
older chrome driver (consider update to the latest chromedriver https://chromedriver.storage.googleapis.com/index.html?path=73.0.3683.68/)
Chrome browser version mismatch (check the browser version and chromedriver compatibility at https://sites.google.com/a/chromium.org/chromedriver/downloads/version-selection)
Consider executing a single #Test directly to make sure the environment is good, then use the testng.xml for further executions.
I use the Surefire Maven plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<argLine>-Dhttp.proxyHost=${http.proxyHost}
-Dhttp.proxyPort=${http.proxyPort}</argLine>
</configuration>
</plugin>
to run multiple JBehave tests with JUnitStories
public class SystemTest extends JUnitStories {
protecetd List<String> storyPaths() {
// returns a list of multiple JBehave files
}
}
and get a single-line report that summarizes the whole test suite:
-------------------------------------------------------------------------------
Test set: <path>.SystemTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.803 sec -
in <path>.SystemTest
Can I configure JUnitStories or Surefire such that I get one line for each file given by storyPaths()?
After trying around with the configuration for a while, I gave up and wrote a custom JBehave reporter that writes multiple JUnit result files, one for each story file.
I'm using JUnit 4.4 and Maven and I have a large number of long-running integration tests.
When it comes to parallelizing test suites there are a few solutions that allow me to run each test method in a single test-class in parallel. But all of these require that I change the tests in one way or another.
I really think it would be a much cleaner solution to run X different test classes in X threads in parallel. I have hundreds of tests so I don't really care about threading individual test-classes.
Is there any way to do this?
Use maven plugin:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.7.1</version>
<configuration>
<parallel>classes</parallel>
<threadCount>5</threadCount>
</configuration>
</plugin>
</plugins>
</build>
From junit 4.7 it's now possible to run tests in parallel without using TestNG. Actually it has been possible since 4.6, but there are a number of fixes being made in 4.7 that will make it a viable option. You may also run parallel tests with spring, which you can read about here
Inspired by JUnit's experimental ParallelComputer runner I've built my own ParallelSuite and ParallelParameterized runners. Using these runners one can easily parallelize test suites and parameterized tests.
ParallelSuite.java
public class ParallelSuite extends Suite {
public ParallelSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
super(klass, builder);
setScheduler(new RunnerScheduler() {
private final ExecutorService service = Executors.newFixedThreadPool(4);
public void schedule(Runnable childStatement) {
service.submit(childStatement);
}
public void finished() {
try {
service.shutdown();
service.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
e.printStackTrace(System.err);
}
}
});
}
}
ParallelParameterized.java
public class ParallelParameterized extends Parameterized {
public ParallelParameterized(Class<?> arg0) throws Throwable {
super(arg0);
setScheduler(new RunnerScheduler() {
private final ExecutorService service = Executors.newFixedThreadPool(8);
public void schedule(Runnable childStatement) {
service.submit(childStatement);
}
public void finished() {
try {
service.shutdown();
service.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
e.printStackTrace(System.err);
}
}
});
}
}
Usage is simple. Just change #RunWith annotations value to one of these Parallel* classes.
#RunWith(ParallelSuite.class)
#SuiteClasses({ATest.class, BTest.class, CTest.class})
public class ABCSuite {}
tempus-fugit offers something similar, check the docs for details. It relies on JUnit 4.7 and you just mark your test to #RunWith(ConcurrentTestRunner).
Cheers
You can check out the open source library - Test Load Balancer. It does exactly what you ask for - run different test classes in parallel. This integrates at the ant-junit level so that you do not have to change your tests in anyway. I am one of the authors of the library.
Also, think about not running them in threads as you may need a process level sandbox. For example, if you are hitting a DB in your integration tests, you do not want one test to fail because another test added some data in a different thread. Most of the times, tests are not written with this in mind.
Finally, how have solved this problem till now?
You can run the tests in parallel using ParallelComputer provided by Junit itself. Here's a small snippet to get you started.
Class[] cls = { TestCase1.class, TestCase2.class };
Result result = JUnitCore.runClasses(ParallelComputer.classes(), cls);
List<Failure> failures = result.getFailures();
This will help when you need to run tests from code as it has no dependencies on Maven or any other build management tools.
Please note that, this will run all test cases in parallel, if you have any dependencies between different test cases it might result in false positives. You SHOULD NOT have interdependent tests anyway.
TestNG can do that (this was my first reflex - then I saw you're already having a lot of testcases).
For JUnit, look at parallel-junit.
Another choice: Punner, a new parallel junit runner and maven plugin. You don't have to change your code, copy it to your pom.xml:
<!-- Disable default surefire based testing -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>com.github.marks-yag</groupId>
<artifactId>punner-maven-plugin</artifactId>
<version>${version}</version>
<configuration>
</configuration>
<executions>
<execution>
<id>test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
Punner can run test methods in parallel, can keep test outputs separately and clean.
Punner will reduce your mvn console outputs, like this:
[INFO] --- punner-maven-plugin:0.9.13:test (test) # ipc ---
[INFO] Punner report directory: /Users/guile/workspace/ipc/target/punner-reports
[INFO]
[INFO] com.github.yag.ipc.IPCTest.testConnectionHandler.............. PASSED
[INFO] com.github.yag.ipc.IPCTest.testSequence....................... PASSED
[INFO] com.github.yag.ipc.IPCTest.testPartialContent................. PASSED
[INFO] com.github.yag.ipc.IPCTest.testResponseContent................ PASSED
[INFO] com.github.yag.ipc.IPCTest.testPingPong....................... PASSED
[INFO] com.github.yag.ipc.IPCTest.testServerClose.................... PASSED
[INFO] com.github.yag.ipc.IPCTest.testServerSideHeartbeatTimeout..... PASSED
[INFO] com.github.yag.ipc.IPCTest.testClientSideHeartbeatTimeout..... PASSED
[INFO] com.github.yag.ipc.IPCTest.testClientSideHeartbeat............ PASSED
[INFO] com.github.yag.ipc.IPCTest.testClientReconnect................ PASSED
[INFO]
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.952 sec, Time saved: 25.919 sec.
Punner produce surefire compatible outputs, you can also get raw log data and a markdown format report from reports directory:
➜ ipc git:(develop) ll target/punner-reports
total 104
-rw-r--r-- 1 guile staff 11K Oct 15 23:07 TEST-com.github.yag.ipc.IPCTest.xml
-rw-r--r-- 1 guile staff 298B Oct 15 23:07 com.github.yag.ipc.IPCTest.txt
drwxr-xr-x 12 guile staff 384B Oct 8 00:50 logs
-rw-r--r-- 1 guile staff 33K Oct 15 23:07 report.md
Punner is my personal project, I written Punner to speed up unit test phase of some other projects such as IPC framework, fine-grained locking, journal service, distributed workflow engine, etc. It saved a lot of my waiting time.
Punner don't support some advanced feature yet. I'm very glad if you could try it and give me some feedback.
You can change your test to be TestNg test in a minute (you just need to change imports), TestNG is the best in parallel testing.
You could try Gridgain that lets you run distribute your tests across a compute grid.
I want to run only a subset of my unit tests, the ones defined by a specific #Category.
So I read several SO questions, such as this one (which is exactly what I am looking for), and also this one.
The solution of my problem seems to be provided by the ClasspathSuite project. So I started to write the NewTest and OldTest interfaces that will define my test categories. Then, I created the AllTests suite:
#RunWith(ClasspathSuite.class)
public class AllTests { }
After that, I created a AllNewTests suite:
#RunWith(Categories.class)
#IncludeCategory(NewTest.class)
#SuiteClasses( { AllTests.class })
public class AllNewTests { }
Finally, I create two JUnit classes, one per category:
#Category(NewTest.class)
public class SomeNewTests {
// some tests...
}
#Category(OldTest.class)
public class SomeOldTests {
// some tests...
}
Now, if I run AllTests, Eclipse launches all the tests of my project, while Maven fails as no test are found:
mvn test -Dtest=AllTests
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running my.company.AllTests
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.093 sec
There are no tests to run.
If I run AllNewTests (which is the correct thing to do, right?), in Eclipse everything is fine (i.e. it only run the tests annoted with #Category(NewTest.class)) but Maven returns an error:
mvn test -Dtest=AllNewTests
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running my.company.AllNewTests
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.125 sec <<< FAILURE!
Results :
Tests in error:
initializationError(my.company.AllNewTests)
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
The exception thrown is the following:
org.junit.runner.manipulation.NoTestsRemainException
at org.junit.runners.ParentRunner.filter(ParentRunner.java:256)
at org.junit.experimental.categories.Categories.<init>(Categories.java:142)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:35)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:24)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:33)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:146)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
at $Proxy0.invoke(Unknown Source)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:145)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:70)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
My question is what I did wrong?
Technical details: Java 6, Maven 3.0.2, JUnit 4.8.1, Surefire plugin 2.7.1, cpsuite-1.2.5
As an update: as of Surefire plugin v2.11, JUnit 4.8+ style categories are now supported.
The release notes for Surefire v2.11 mention the new feature. The surefire:test goal can is configured using groups.
I have solved my problem by creating my own JUnit Runner, that extends the Suite.
The idea is close to the principle by the Classpath Suite project, i.e. looking for the classes existing in the classpath, and keep only the ones that are annotated with a given annotation (#NewTest for example).
If you are interested, you can read the full story on my blog.
After reading some blog posts and stackoverflow questions, I finally was able to make this work with the surefire plugin, as user1034382 answered. In my case with version 2.17 of maven-surefire-plugin.
Just to add my two cents, the more up-to-date explanation can be found here:
Using JUnit Categories to group tests
But you may run with the following surefire plugin problem:
[ERROR] java.lang.RuntimeException: Unable to load category:
That can be fixed with this other stackoverflow question/answer:
Where should I put interface class for Junit #Category?
My answer is just to gather all these info here and avoid googling/reading to many diferent solutions. Al least, this worked to me.