Arquillian Core Suite/SubSuite support


#1

Core issue ref:

It’s taken years, but the most requested feature is only centimeters away!

For the last week or so I’ve been hacking out the Suite support in Arquillian Core. I believe I’ve come to a fairly non intrusive solution: https://github.com/aslakknutsen/arquillian-core/tree/arq_suite

Essentially it consist of 4 changes/additions:

  • SubSuiteScope
  • Instance<T>.all() to get Collection<T> of all T in active scopes
  • TestClass represent active Class or Classes(children) depending on which level we’re on
  • JUnit Arquillian.class is now delegating to either SuiteRunner or the old TestRunner.

How it works

@Suite(value = "org.jboss.arquillian.showcase.universe*|.*(core|warp|drone|graphene|persistence).*Suite", strategy = PackageResolverStrategy.class)
@RunWith(Arquillian.class)
public class ApplicationSuite {

   @Deployment
   public static WebArchive deploy() {
       return Deployments.Client.full();
   }
}

The Arquillian runner will look at the testClass and see if it has a @Suite annotation on it, if it does it will delegate to the ArquillianSuiteRunner, if not it delegates to the ArquillianTestRunner(old Arquillian).

The Suite annotations takes a value of String[] and some form of ResolveStrategy:

public interface ResolveStrategy {

    List<Class<?>> resolve(String[] values);
}

Currently we have 2 ResolveStrategies; ClassResolverStrategy and PackageResolverStrategy.

ClassResolverStrategy

Simple enough it attempts to load the Suite.value String[] as fully qualified Classes via the ClassResolverStrategies ClassLoader and does a new resolve from the top of he found class. Each loaded Class will be a Child of the @Suite annotated class.

PackageResolverStrategy

Package resolver uses the ShrinkWrap URLPackageScanner to scan for certain classes and match them against the given expression. The @Suite.value expression is as follow:

"package.name.to.scan"[*]|"reg-exp-to-match-found-classes"

Example:

"org.my.test.suite*|.*TestClass"

Adding a ‘*’ to the en\d of the package name tells the scanner to perform a recursive package scan from the given package. Without the ‘*’ only the given package is scanned.

ArquillianSuiteRunner

Since the resolved classes go through the same chain as the initiating class we support nested Suites with Children on any level.

MySuite
   - ChildTestClass
   - MySubSuite
      - Child2TestClass

A new level has been introduced in the SuiteEvent hierarchy: Suite -> SubSuite -> Class -> Test

Along with that comes two new Lifecycle events as well; BeforeSubSuite and AfterSubSuite

The SubSuite scope is a bit different then the previous scopes as they are applied chained up to the parent level per event. So assuming the we’re running the Test Child2TestClass from the example above; Both SubSuiteScope(MySuite) and SubSuiteScope(MySubSuite) will be active during execution.

Collection<T> Instance<T>.all()

During BeforeSubSuite event the current TestClass will go through a similar process as BeforeClass, firing GenerateDeployment and DeployManagedDeployments events causing a DeploymentScenario being added to the current SubSuiteScope based on the given Suite class.

The Suite class does not currently support Lifecycle methods or Test methods, but basic metadata will be read, e.g. @Deployment definitions.

The DeploymentScanario.deployment(DeploymentTargetDescription) now use the new Instance<DeploymentScenario>.all() method to look for Deployment’s in all active scopes. It currently does not do any validation of duplicate named Deployment’s in the parent chain, only on a per Class scan basis. If MySuite and MySubSuite both define a default or named deployment with the same name, the deployment closest to the running TestClass win. e.g. Child2TestClass will run in the deployment defined in MySubSuite if MySubSuite override a Deployment from MySuite.

TestClass

The TestClass now hold the parent/child relationship to represent the Suite/SubSuite classes. TestClass in MySuite will have getJavaClass() MySuite.class but with getChildrenChain() [MySuite.class, ChildTestClass.class, MySubSuite.class, Child2TestClass.class]. Similar MySubSuite will have getParent() MySuite.class.

Extensions

The current implementation is intended to be backward compatible with all extensions as long as they are not ran as part of a Suite. The only exceptions as far as I can tell are QUnit and Spock as they extend the Arquillian.class directly: Arquillian.class is now just a delegate and any overridden methods won’t work. The old Arquillian.class is the new ArquillianTestRunner.class. The Runners factory method will currently create an ArquillianTestRunner regardless of what the @RunWith on the found class says. Some more work is needed here.

What works with Suite

Arquillian Drone :ike_success:

Drone seems to run fine out of the box, no modifications needed.

Arquillian Graphene :ike_success:

Graphene seems to run fine out of the box, no modifications needed.

Arquillian Warp :ike_warning:

Works mostly out of the box given the Suite class where the Deployment is defined has a @WarpTest annotation.

@WarpTest
@Suite(value = "org.jboss.arquillian.showcase.universe*|.*(core|warp|drone|graphene|persistence).*Suite", strategy = PackageResolverStrategy.class)
@RunWith(Arquillian.class)
public class ApplicationSuite {

   @Deployment
   public static WebArchive deploy() {
      ... 
  }
}

But Warp does not play nice with other TestClasses that want to run incontainer. With a single Deployment/single TestClass setup Warp had complete control and could assume that ‘anything’ being called against the given Deployment had intentions of going via the WarpFilters, but with a Suite that is no longer the case. Meaning Mixed Run mode with Warp is broken. See https://issues.jboss.org/browse/ARQ-1935

Arquillian Persistence :ike_error:

Persistence extension rely on scanning the TestClass annotations to bundle the given data scripts, this does not work with a Suite as the scanned Suite class does not contain any @UsingDataSet/@ShouldMatch related annotations. You can get Persistence extension to trigger by adding @PersistenceTest annotation to the Suite class, but you will need to manually package the datasets in the Deployment where Persistence extension is configured to look for them.

@PersistenceTest
@Suite(value = "org.jboss.arquillian.showcase.universe*|.*(core|warp|drone|graphene|persistence).*Suite", strategy = PackageResolverStrategy.class)
@RunWith(Arquillian.class)
public class ApplicationSuite {

  @Deployment
   public static WebArchive deploy() {
       return Deployments.Client.full()
             .addAsResource("datasets/conference_with_speaker.yml", "datasets/conference_with_speaker.yml");
   }
}
Arquillian Transaction :grey_question:
Arquillian Spock :ike_error:

Arquillian.class is no longer the same Class. Direct override won’t work.

Arquillian QUnit :ike_error:

Arquillian.class is no longer the same Class. Direct override won’t work.

Arquillian Jacoco :ike_success:
Arquillian Byteman :grey_question:
Arquillian Rest :grey_question:
Arquillian Recorder :grey_question:
Arquillian Cube :ike_success:
Arquillian Droidium :grey_question:
Cuke in Space :grey_question:
Container: JBoss AS 7 / WildFly :ike_failure:

These adopters does currently not work due to assumption about a active ClassScope during deploy. They should work when used via the arquillian-container-proxy as the additional observers causing the problems are not loaded: https://github.com/aslakknutsen/arquillian-container-proxy (not tested)

See Arquillian Core Suite/SubSuite support

Extension summary

In general, any extension that only reads metadata from the TestClass runtime and does not effect the packaging process based on that metadata should run fine with the Suite. Extensions that do read metadata from the TestClasses to either effect the packaging processes with config data or to activate it self might require some update to support the new suite.

As part of https://issues.jboss.org/browse/ARQ-472 we should add some Metadata querying to the TestClass and apply the query to all children of the given class. That would allow an extension to find all fields/methods with given annotations across the whole Suite.

e.g.

public boolean activateWarp() {
   return testClass.query().forClass().withAnnotation(WarpTest.class).size() > 0
}
public void addDataSets() {
   for(UsingDataSet config : testClass.query().forMethods().withAnnotation(UsingDataSet.class)) {
     deployment.addAsResource(load(config.value()))
   }
}

The running example above can be found in the arquillian-showcase repository in the universe_suite branch: https://github.com/arquillian/arquillian-showcase/tree/universe_suite

The latest deployed 1.1.8.Final-SNAPSHOT is based on the arq_suite branch: https://github.com/aslakknutsen/arquillian-core/tree/arq_suite

Please take it for a spin;

  • report back extensions that doesn’t work
  • odd behavior
  • and general improvement feedback

Arquillian Core Suite/SubSuite: How to define the Suite
Community hangouts
#2

Forgot to add, there is an assumption in the JBossAS7/WildFly adapters that when a Deployment happens there is a Class scope, that has for mostly been true so far, tho not by design. To workaround that you can use the following temp extension fix:

import org.jboss.arquillian.container.spi.event.container.BeforeDeploy;
import org.jboss.arquillian.core.api.annotation.Observes;
import org.jboss.arquillian.core.spi.EventContext;
import org.jboss.arquillian.core.spi.LoadableExtension;

public class TempFixExtension implements LoadableExtension {

    @Override
    public void register(ExtensionBuilder builder) {
        builder.observer(IgnoreWFStartupServiceFailure.class);
    }

    public static class IgnoreWFStartupServiceFailure {
        
        public void ignore(@Observes EventContext<BeforeDeploy> ec) {
            try {
                ec.proceed();
            }
            catch(NullPointerException e) {
                // Error in WF ServerSetupObserver, assumes an active ClassContext
            }
        }
    }
}

#3

Personally, I think if you’re using @Suite, you should not have value and instead have packageRegex/classRegex to define the resolution, or define different annotations for each.

I’m assuming that this will automatically package any test dependencies in suite as long as annotated.

One thing that’s not clear (and similar issue with the existing suite extension) is if the suite class should contain any test methods, or can it, or if other tests need to reference it.

How would I run from the CLI if I wanted to only run one test.


#4

Currently no Test or Before/After is executed on the Suite class. It’s essentially just a redirected Deployment Scenario definition that is reusable across multiple Test Classes.

Similar to how you would run a normal JUnit suite; either define include/exclude filters in Surefire or using the ‘test’ option. That’s one of the down sides described here; Arquillian Core Suite/SubSuite: How to define the Suite


#5

I came across an issue while playing around with a simple test-suite using @ Suite and @ BelongsTo annotations. It ends up in an java.lang.IllegalArgumentException: No active SubSuiteScoped Context to bind to.

TestSuite:

@ RunWith(Arquillian.class)
@ Suite(value = "org.arquillian.example|Gree.*", strategy = PackageResolverStrategy.class)
public class **AllTests** {
  @ Deployment
  public static JavaArchive createDeployment() {
    JavaArchive jar = ShrinkWrap.create(JavaArchive.class).addClasses(Greeter.class, PhraseBuilder.class).addAsManifestResource(EmptyAsset.INSTANCE, "beans.xml");
    return jar;
  }
}

Single Test:

@ RunWith(Arquillian.class)
@ BelongsTo(value = AllTests.class)
public class GreeterTest {

  @ Inject
  Greeter greeter;

  @ Test
  public void should_create_greeting() {
    Assert.assertEquals("Hello, Earthling!", this.greeter.createGreeting("Earthling"));
  }
}

What kind of SubSuiteScoped Context is expected here? Any help appreciated.


#6

Not tested much by using both strategies at once. Wondering if you might end up in some kinda wired loop here since the Child Strategy point to a Scanner strategy.

Try removing the @Suite and @RunWith on the AllTests class, then run GreeterTest as a normal test. The Junit Runner should figure out that multiple ‘Children’(GreeterTest) belong to the same ‘Suite’(AllTests) and run them ordered.


#7

I am still trying to get my tests up and running. After eliminating @ Run and @ Suite I am running into a NullPointerException while starting up my test:

java.lang.NullPointerException
at org.jboss.as.arquillian.container.ServerSetupObserver.handleBeforeDeployment(ServerSetupObserver.java:84)

There is no further caused by issue in the stack trace.


#8

See Arquillian Core Suite/SubSuite support

Using Arquillian Chameleon instead of the WildFly adapter directly should handle this as well (same as referred to as arquillian-container-proxy);


#9

I already tested against several containers using arquillian chameleon. Every time it ended up in the same error (NullPointerException) which occurs ways before the actual tests get started.

Maybe I will set up a new environment and give this whole thing a fresh start in the next days. Do I still have to rely on your repo or is it sufficient to use 1.2.0.Alpha1 of https://github.com/arquillian/arquillian-core?


#10

This looks like something we desperately need because we have flaky tests caused by problems un-/re-deploying parts (contained wars) of a rather a big ear with our smoke tests in combination qith Arquillian Sputnik and Spock. The biggest part of the ear is always the same though, so it would be nice to just deploy it once.

For one years the feature has been centimeters away. We use Arquillian Core 1.11.1. Can we expect this one to be included into the product anytime soon?


#11

Is there any intention to release this? Or is this project abandoned?


#12

Sure there is, Arquillian is alive like never before. It’s just not a trivial change and requires careful refactoring.

We will keep this post active with the updates on this feature.


#13

I see that there’s no development on the branch initiated by @aslak since 2 years ago and no sign of it in 1.2.0.Alpha1 as announced it would happen since 1st of January 2016. Is there anything scheduled? An approximation of when this will be integrated in the main branch?