Arquillian Core Suite/SubSuite: How to define the Suite

[aslak]

Continuing the discussion from Arquillian Core Suite/SubSuite support:

We need to figure out how to expose the Suite as an API.

Definitions

• Child: A Test Class
• Metadata: The definition of Deployments and other configuration
• Suite: A Collection of children sharing the same metadata
• SubSuite: A hierarchical level within a Suite

There's more than one way to skin a cat.

Feel free to comment, propose or 'like/vote' on the individual styles as you see fit.

What works for you?

[aslak]

Suite Scanner

@Suite(value = "org.jboss.arquillian.showcase.universe*|.*(core|warp|drone|graphene|persistence).*Suite", strategy = PackageResolverStrategy.class)
@RunWith(Arquillian.class)
public class ApplicationSuite {

   @Deployment
   public static WebArchive deploy() {
       return Deployments.Client.full();
   }
}

ApplicationSuite
   - ChildTestClass
   - ApplicationSubSuite
      - Child2TestClass

The current proposal fairly closely matches how normal JUnit Suite works, by adding multiple children to a given Suite level recursively. One thing Arquillian has that normal JUnit doesn't have is a dependency on the Suite level it self, e.g. Deployment definitions or hierarchical metadata.

Pros

• Children could be included in different suites
• The parent can define completely how to run found children, e.g. no need for @RunWith on child
• Multiple suites in one module possible

Cons

• Complexity in defining the Suite scanner, the strategy
• Loosing the ability to run the child standalone as it has no definition of it's parent outside of what the parent says
• Need special build config to only run the Suite classes

Implemented in 1.1.8.Final-SNAPSHOT

[aslak]

Child reference

public class ApplicationSuite {

   @Deployment
   public static WebArchive deploy() {
       return Deployments.Client.full();
   }
}

@BelongsTo(ApplicationSuite.class)
public class ApplicationSubSuite {

   @Deployment(name = "backend")
   public static WebArchive deploy() {
       return Deployments.Client.full();
   }
}

@BelongsTo(ApplicationSuite.clas)
@RunWith(Arquillian.class)
public class ChildTestClass {
   @Test
   public void t() {}
}

@BelongsTo(ApplicationSubSuite.clas)
@RunWith(Arquillian.class)
public class Child2TestClass {
   @Test
   public void t() {}
}

The Child could point to where the Metadata about it's deployments are defined, if multiple Children in the current run point to the same metadata, they are considered to be a 'Suite'. A Suite of one or many, just depends on whos pointing to what. Not 100% sure how to map that physically in a JUnit Suite UI style as I think we technically could get 3 Class requests and some how have to fool the IDE to see that as 1 node with 3 children.(But that's another issue)

Pros

• Running a single child via IDE
• Multiple suites in one module possible
• Run order irrelevant

Cons

• Need to define @RunWith on children
• Only one Suite per Child (could make a Base Child class and have multiple extended classes point to different Suites)

Implemented in 1.1.8.Final-SNAPSHOT

[aslak]

Class inheritance

public class ApplicationSuite {

   @Deployment
   public static WebArchive deploy() {
       return ShrinkWrap.create(WebArchive.class);
   }
}

@RunWith(Arquillian.class)
public class ChildTestClass extends ApplicationSuite {

   @Test
   public void t() {}
}

@RunWith(Arquillian.class)
public class Child2TestClass extends ApplicationSuite {

   @Test
   public void t() {}
}

Pros

• Easy to understand
• Follow normal Java rules/scopes
• Running a single child via IDE
• Run order irrelevant

Cons

• Single suite per Child
• Forces Java inheritance
• Inflexible
• Not backwards compatible as it could have side effects on an old test suite (could be fixed with e.g. a forced @Suite on super class to trigger the behavior)

[aslak]

Global module Suite

<arquillian>
   <property name="suiteConfiguration">org.some.DeploymentConfiguration</property>
</arquillian>

public class DeploymentConfiguration {

   @Deployment
   public static WebArchive deploy() {
       return Deployments.Client.full();
   }
}

@RunWith(Arquillian.class)
public class ChildTestClass {
   @Test
   public void t() {}
}

@RunWith(Arquillian.class)
public class Child2TestClass {
   @Test
   public void t() {}
}

Allowing a Global Configuration of where to read the metadata, similar to how the Suite Extension works today.

Pros

• Easy to understand?
• Running a single child via IDE
• Run order irrelevant

Cons

• All or nothing
• Only a single suite per module/run

[aslak]

Named reusable deployments

Leave Deployment creation as is, but allow to configure specific deployments to act as 'global' deployments for reuse between multiple Children.

<arquillian>
   <property name="reuseNamedArchives">app.war</property>
</arquillian>

@RunWith(Arquillian.class)
public class ChildTestClass {

   @Deployment
   public static WebArchive deploy() {
       return ShrinkWrap.create(WebArchive.class, "app.war")
                .addClass(Y.class);
   }

   @Test
   public void t() {}
}

@RunWith(Arquillian.class)
public class Child2TestClass {

   @Deployment
   public static WebArchive deploy() {
       return ShrinkWrap.create(WebArchive.class, "app.war")
                .addClass(X.class);
   }

   @Test
   public void t() {}
}

Pros

• Only configuration change needed in API
• Running a single child via IDE
• Run order irrelevant

Cons

• Unpredictable which 'win' when multiple with same name. First serve.
• Hard to make Suite levels visualized in TestRunner
• Require upfront parsing of all Children Deployments to determine who belongs to which deployment

[lordofthejars]

For me first approach is a bit complicated, in my case I needed to read other strategy to then understand exactly what this was about. Also an small mistake in reg exp makes something to work or not which makes devs starting to inspect what is happening if the test is not defined correctly if it is the regular expression, if they have added something in regular expression that it was not necessary...

Also I think that if you cannot run the child alone it can be a hassle when one test of the whole suite doesn't pass.

The global and named approach I will discard because I like to have as much information as possible in test so with a quick overview I can know exactly what the test does or where belongs. So having to configure it in another place makes any developer to remember that maybe in arquillian.xml there is something that may be important. But reading both approaches I would prefer GlobalModule instead of Named because at Global at least you see that there is no @deployment and probably it is because the definition is on arquillian.xml.

And then we have Class inheritance. Maybe it is the most natural one choice but during my career I have seen a lot of tests that extends from some custom class because they have implemented there some custom operations or extensions. I know that this can be considered a bad practice in some places, but if we simply cut the wings to people who are doing this from his point of view will mean that we are not helping them at all.

So although it may be the most complicated ones for implementing Child reference would be the easiest one to explain and use it.

[robertpanzer]

In my opinion having the ability to start a single test in the IDE is a must.
So SuiteScanner is out imo.

Named resusable components has the problem that it enforces different application names if you have multiple suites in one project and some applications (I know at least one;-)) relies on JNDI names of the form java:app//...

Global module suite will place constraints on the project layout and I personally don't like that very much.

Class inheritence has the problems Alex stated and a single suite per project is the same problem that I didn't like so much at Global module suite.

So I would agree with Alex and prefer Child reference.
Maybe the restriction of one test only belonging to one suite could be circumvented by placing the test methods into super classes that the concrete test classes extend.
The concrete test classes would do nothing more than inherit the test methods and point to the test suite they belong to:

public abstract class AbstractTest {
   @Test test() {...}
}

@RunWith(Arquillian.class)
@BelongsTo(ApplicationSuite1.class)
public Test1 extends AbstractTest {}

@RunWith(Arquillian.class)
@BelongsTo(ApplicationSuite2.class)
public Test2 extends AbstractTest {}

[aslak]

Maybe the restriction of one test only belonging to one suite could be circumvented by placing the test methods into super classes that the concrete test classes extend.

Yeah, that's the workaround I was thinking about as well.

[Kolio]

lordofthejars and robertpanzer motivated it pretty well(including the workaround).
+1 for child inheritance.

[aslak]

@Kolio I assume you mean Child Reference ?

[aslak]

Both Suite Scanner and Child Reference are implemented in the latest published 1.1.8.Final-SNAPSHOT version.

For Suites to work in Maven you need to force Surefire to use the JUnit 4.7 provider:

<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-surefire-plugin</artifactId>
			<version>2.18.1</version>
			<dependencies>
				<dependency>
					<groupId>org.apache.maven.surefire</groupId>
					<artifactId>surefire-junit47</artifactId>
					<version>2.18.1</version>
				</dependency>
			</dependencies>
		</plugin>
	</plugins>
</build>

[aslak]

4.4 seconds to run 238 integration tests.. not too shabby

[miklosovic]

I am for child reference as well.

I do not know if "BelongsTo" annotation is the best one. Not saying it is chosen badly, just that the annotation alone does not quite say it has something to do with Arquillian suite. If I saw it for the first time, I would not know that "BelongsTo" has some relationship to suites at all ...

[aslak]

I do not know if "BelongsTo" annotation is the best one.

Agree. Not sure if we should describe that this deployment scenario is read from somewhere else (and since many read from this they can reuse) or if we should be more explicit and describe it as this is part of the larger suite definition x..

[miklosovic]

Maybe just putting the word "suite" into that annotation in order to be more explicit could do the job.

@BelongsToSuite(MySuite.class)

@PartOfSuite(MySuite.class)

@SuiteAware(MySuite.class)

If the only intent of suite class is the deployment providing, that annotation could be maybe more deployment-centric. Something like @DeploymentProvider(MyDeployments.class) or anything like that, just an idea.

[robertpanzer]

I also like these as they better communicate what the annotation is provoking

@BelongsToSuite(MySuite.class)
@PartOfSuite(MySuite.class)

@SuiteAware is not a favorite of mine because it suggests that it would also run without.

@DeploymentProvider would also not really hit the whole point if a suite would also have callbacks @BeforeSuite and @AfterSuite in the future or if a suite could contain rules.

[Kolio]

Sure I meant Child Reference - it got mixed up. But I see it is already implemented - thanks.

[PeterProbst]

I vote also for child reference. What about the following for referencing more than one suite:

@BelongsToSuite({MySuite.class, SecondSuite.class})

The point from robert is for us very usefull:

@BeforeSuite and @AfterSuite

When this works in combination with persistence extension api:

@UsingDataSet and @DataSource

then we can inject data for all tests on different datasources. Very cool feature...
So for this to work well the

@BeforeSuite and @AfterSuite

Annotation should not be restricted to use only once in suite. If annotated on more than one method you can use same/different datasets on different datasources. Very useful in our case, as we have a multi schema application...

[ochaloup]

Hello,

just for my interest. Are there some plans for merging the suite functionality to new release of core?

Thank you
Ondra