I am working in an enterprise with some legacy applications, which are quite coupled.
Integration test is thus very important. I am looking for a distributed testing framework, or a combination of them that QA team and development team can both leverage on. For "distributed" I refer to the many environments for testing, while a single application itself is not a distributed system.
The following criteria might not be the ideal model, but kind of requirement and expectation of the tool to be developed.
Such framework should be :
open sourced
community support to learn, maintain and extend :)
distributed
allow tests to be execute in multiple environments.
since test cases may change quite often, it should be managable to distributed those test into target machines with target codebase to run.
I am checking out [STAF][1] on this.
allow integration tests
I am looking into Camel testing. we have some existing "simulator", "mock" to stimulate some applications behavior.
framework with easy-to-use dependency injection / mock will be very helpful..
with WEB UI, wich is a single place that can
- all testing results are shown
- able to trigger test
- able to see testing definition
Human readable and configurable acceptance testing definition
Testing language should be in a readable DSL
I wonder is framework like FitNesse a good choice as I did not figure out a way to test it distributed.
For Citrus I am concerning XML is hard to read
Some current frameworks
My team has been suggesting JSystem, while it is possible to distribute tests, there is no web GUI and
my major concern is its development has been stopped since 2009
I looked into many frameworks as mentioned,
Grinder - seems better suite web load testing?
JMeter- this seems to be closer, but also focused on load test?
more on background: most developers are familiar with JAVA and currently we use Hudson to run JUnit tests.
And finally I am also thinking a more conceptual issue, should xUnit test, integration test, acceptance test all being separated? xUnit test should be stick with code base? perhaps this worth another question, but with that in mind I am not sure I am looking for the right thing.
I will greatly appreciate if you can give me some comments on the thought / suggest some frameworks. Thanks a lot in advance.
I will suggest you to use Jenkins Framework.
Open source
Can be used for different environments
I myself working over Integration Testing using Jenkins so suggest you for this.
Web UI is very interactive.
Large variety of plugins is available and development is still on.
Write some ANT, MAVEN scripts you can get your job done through this.
Related
I am working on a J2EE web-application. I would like to upgrade some of major open-source libraries from older version to newer one based on availability of libraries on maven repository.
For example of libraries : Google-Sitebricks, Jetty, JSON, Jackson, JUnit, HttpClient etc.
What would be the best way to ensure that the after upgrades, there are no side-effects/leakages happened in application.
Here, I tested the application's basic functionalities like sanity test to ensure its working or breaking the things.
I am wondering on what is the standard process need to be followed in such scenarios. Please suggest the effective & best approaches/practices for this.
Thanks
Test, test, and test. And then test some more.
Hopefully you have automated tests which will shake out problems. If not, you will find yourself spending a lot of cycles manually testing. Just sanity checks may not be enough; version upgrades can break functionality in subtle ways.
Even before you get to the testing, you may find that your application fails to compile after upgrading the libraries.
In any case, I would not upgrade many libraries all at once, except when upgrading a library causes a cascade of upgrades in transitive dependencies. Upgrading them one by one is the safest thing to do.
I was hoping you could point me in the correct direction. I am trying to find an automation tool for an internal Chrome site that I, as a tester, can use to create tests. I would prefer some kind of recording type option and free or very low cost. And it MUST work on Chrome. Our app doesn't work with any other browser
Here's what I've looked at so far:
I am not going to have time to learn any new languages or have access to a developer, so Cucumber is out (this was the company's first choice).
I found the Ghost add-on that records, but it needs to open the firewall to be used. BIG no no.
I looked at sikuli, but we can't have a dedicated screen like that, although it might still be a last ditch option
TestComplete might work, but it costs money. I am going to download the free trial, but the justification may not be enough to get the company to spend the money
SoapUI is something I've worked with before and I know that only the Pro is helpful to a non-programmer like myself.
Any ideas are welcome.
Thanks!
You could look at Robot Framework. It is a keyword driven test automation framework and it has good Chrome support via Selenium. Web page has examples how to write tests for browsers.
Robot Framework is open source and free. Writing tests with keywords looks more difficult than using record-playback tool. However, once you have several tests, managing them will be easier with keywords.
I'm new to AS3 and Flex, but I am creating a mobile app with them in FlashDevelop. I want to add some unit tests, but I can't find any really good frameworks or documentation on it. I saw a little bit on FlexUnit and AsUnit, but both seem to be a few years old, and there isn't much documentation on using them in FlashDevelop.
Is there a good framework I can use for unit testing here? Even if I can't test the UI layer, I'd like to at least make sure my backend functions work correctly.
If you are testing backend functions, then you generally want to use the testing framework of whatever backend you are using. If you want Flex to test your backend functions, then you aren't Unit Testing, you are Integration Testing. While FlexUnit seems old, it still performs well at serving its purpose. The usage of FlexUnit should be independent of your IDE, you may just have to do more typing.
You can try to use the Flexunit4.0 for testing the business logic on the client side mixing it with the Mockito. None of these actually support testing the UI.
i've never tried to test the backend services with the Unit testing because of the authentication issues, but we created another testing UI for testing the services, This UI has a tons of screens, each screen to test different services.
I've seen that there is a ServiceBrowser for amfphp, i'm not sure if this can be used for Java or other platforms.
Other ways of testing the UI is by automation testing, use flexmonkey, selenium for that.
Disclaimer: I am relatively unfamiliar with the flash build processes, so some/all of this may be misinformed nonsense. Please feel free to suggest alternative approaches.
We're currently developing a flex web app and our build situation is far from ideal. At present we're (as in individual developers) just building using FlashBuilder and deploying manually. The programmers are currently screaming bloody murder for two reasons, though:
The lack of CI is like going back to the stone age
We don't much care for FlashBuilder
(Note: We're only using FlashBuilder because it was the easiest way to set up a flex project in conjunction with Away3d and get it building / rendering correctly -- it's a stopgap solution).
As a predominately .NET development shop, we're used to doing continuous integration as well as continuous deployment. Ideally, we'd like to get something comparable to this for our flash projects without tying ourselves to a particular IDE.
Requirements:
The build process must be:
.. runnable via the commandline
.. runnable on both developer and CI build machines (and certainly not requiring an IDE!)
.. preferably as IDE-independent as possible (pragmatism will kick in though; if this causes a lot of friction we'll just pick one).
.. able to run on Windows (we develop using Windows)
We don't mind a touch of duplication or a few manual steps (e.g. tarting up the build scripts if we add a new project via an IDE, or generating one configuration from another if tools exist), but the less duplication / maintenance required the better.
I've read quite a few articles / blog posts and watched some short screencasts, but most of them are very thin on the ground on how the build system sits alongside IDEs. Most articles/screencasts have the same formula: How to create a "Hello World" build using a single file & text editors (no IDE).
I've not seen the topic of multiple libraries/projects etc. being broached, either.
After reading around the issue for a while, I'm considering investigating the following options:
Project Sprouts
Flexmojos
Maven Flex Plugin
buildr as3
Does anyone have any experience of the above solutions (or others I'm unaware of) and, if so, what do you make of them? Any help / pointers appreciated.
I recently started building with Gradle and the GradleFx plugin and I immediately fell in love with its power and ease of use.
Gradle is ANT + Maven + Ivy evolved and is primarily used from the command-line. You can:
write scripts in Groovy (a powerful Ruby-like language that runs in the Java Virtual Machine)
access all existing Maven and Ivy repositories as well as your own repos
use existing ANT tasks
integrate with CI (in Jenkins you just tick a checkbox to activate Gradle support)
although it has originally grown from the Java/Groovy community, it is in fact language agnostic. You add language-specific plugins for added functionality. GradleFx is such a plugin that provides you with additional ActionScript/Flex building tasks.
do easy multi-project builds. e.g. you can compile, unit test, package and deploy both your .NET service layer and your Flex client application with just one command.
use convention over configuration: if you stick to the conventions, your build scripts will be extremely terse
generate all kinds of reports: unit testing, checkstyles, codenarc, ...
generate Eclipse, IDEA or other IDE projects
all the things I haven't discovered yet
And best of all: it's very easy to learn. I had no knowledge of Maven before I started with Gradle and could get a multi-project build with some customizations working quite quickly.
Edit (comparison to Buildr AS3 and Maven)
I can compare this only to one of the projects you mentioned: Buildr AS3. It seems to start from a philosophy that is similar to Gradle's. But I've tried to use it about half a year ago and couldn't even get a simple 'Hello World' app to work. I e-mailed the developer for help: no response.
Compared to GradleFx: I had a small forum discussion with the developer (on a rather philosophical topic, since I didn't really need any help because it just worked right away). He answered within minutes.
As for Maven: (for what it's worth) I've only glanced at some configurations and they seem overly complicated when I compare them to a Gradle script.
There is one thing Maven does that you can't do with GradleFx (yet). It can pull the right Flex SDK from a Maven repo and build against that. With GradleFx you have to have your SDK's available locally.
I'm quite familiar with using maven as the main build tool and the flexmojos plugin from Sonatype. My experience has been a bit of a roller coaster with flexmojos. Maven is completely solid, it works all the time without issue, the only issue is the flexmojos plugin which has fluctuated a lot between versions. If you choose to go this route make sure to grab the source for flexmojos so you can see what your configuration options are actually doing to the command line parameters etc. For Flex 3.x flexmojos 3.x up to around 3.9 is good and works fine with regard to the goal for generating the .project eclipse files, believe there's also a mojo (a maven plugin) for generating intelliJ IDEA project files as well as others. If you're using Flex 4 you can compile with the latest flexmojos 4.0RC2 but it appears to me that the goal for generating flex/flashbuilder project properties is now gone (I'm not sure if this is because it's been replaced by another plugin altogether or what the deal is). However building with maven and flexmojos does fulfill all of your goals above (we also use it for building our service layer, so in a single mvn clean install we get a jar packed in a war packed in an ear with everything configured and a swf, that part is really nice). Also you can do continuous integration using bamboo (or simply write your own script that is triggered from a cron job or in windows as a batch file executed with a scheduled task if you don't have a *nix server around). Let me know if you'd like any more details or if I missed something major.
Shaun
I have been using Hudson, now Jenkins, with Ant for Flex automated builds and FlexUnit testing. Jenkins has some really useful plugins for integration with eclipse (and hence, FDT or FlashBuilder), Jira, SVN, Git etc., and it's free. Also, you can integrate the Ant build into Maven scripts, so I've found this to be a good and flexible solution for all purposes I've come across so far.
The Flex SDK comes with Ant tasks, and writing even elaborate Ant build scripts is quite easy - in fact, I'd been using Ant locally before, and I could reuse my existing scripts with only a few added extra compiler options for FlexUnit tasks.
However, it took a while to set up the system correctly for unit testing, because I'm running a headless server on Linux, and that implicates a rather complicated environment for ActionScript tests, because they run only in Flash Player. This, of course, is true for all CI scenarios using FlexUnit, regardless of which server you use.
Here's what I've learned:
FlexUnit needs a standalone debugger version of Flash Player installed, but Adobe only distributes binaries for the standard version on Linux. Therefore, compiling from source was necessary, and since my server system is stripped down to the bare necessities, it took some effort to install all the correct dependencies and get them to work.
The Flash Player needs hardware to run correctly: It uses graphics, therefore it needs a graphics card, and sounds, therefore it needs a sound card. On my headless server, this meant I had to install a VNC host to get it to run at all, and I had to eliminate any tests using sounds (those will now only run on local machines). If anyone ever comes across a working sound card emulation for openSuSE that I could use with the VNC client - you'll be my hero forever!
If you've set asynchronous timeouts in your unit tests, and/or you need to use setTimeout() to send delayed procedure calls, make sure the intervals aren't too short - I've had problems with tests that ran fine on any local machine, but broke the build on the CI server, because the Flash Player is considerably slower on the VNC client than on an actual graphics card.
I've also found this last issue to be a healthy lesson: Criteria for unit tests should not be based on assumptions about the system's performance, or at least be tolerant enough to succeed even on a slow machine.
Do most people use there selenium (in my case selenium 2 with web driver) tests with JUnit? I personally do not and was wondering if I am in the minority. I tend to write all my own classes and exception handling and have everything write to a database for reporting on results.
The only problem I see with your approach is that the more code you write yourself instead of relying on existing third party libraries, the more maintenance you will have.
If you have the option of using third party, your should (in most cases) integrate it instead of writing something yourself.
If you want a testing framework but feel like JUnit is too restrictive, you might want to consider using TestNG. It is designed for integration testing and has features that make GUI automation easier.
At the end it comes to your personal choice,if your test code is small and easier to maintain and keep track of.For enterprise-level web application having multiple modules to be automated,we have to choose some sort of test framework for easier automation and reporting purposes.
I personally prefer TestNG as it is code is clutter free and more readable.Cheers!