The first tests you might want to write are the ones which concern parsing.

This reflects the fact that the grammar is the first thing you must write when implementing a DSL. You should not try to write the complete grammar before starting testing: you should write only a few rules and soon write tests to check if those rules actually parse an input test program as you expect.

The nice thing is that you do not have to store the test input in a file (though you could do that); the input to pass to the parser can be a string, and since we use Xtend, we can use multiline strings.

The Xtext test framework provides the class ParseHelper to easily parse a string. The injection mechanism will automatically tell this class to parse the input string with the parser of your DSL. To parse a string, we inject an instance of ParseHelper<T>, where T is the type of the root class in our DSL's model—in our Entities example, this class is called Model. The ParseHelper.parse method will return an instance of T after parsing the input string given to it.

By injecting the ParseHelper class as an extension, we can directly use its methods on the strings we want to parse.

The Xtext generator already generates a stub class in the .tests project for testing the parser. In the Entities DSL, this Xtend class is called EntitiesParsingTest. This stub class is generated for the initial "hello" grammar, so if you run it as it is, the test will fail.

Thus, we modify the stub class as follows:

@RunWith(XtextRunner)
@InjectWith(EntitiesInjectorProvider)
class EntitiesParsingTest {
  
  @Inject extension ParseHelper<Model>

  @Test
  def void testParsing() {
    val model = '''
      entity MyEntity {
          MyEntity attribute;
      }
    '''.parse
    
    val entity = model.entities.get(0)
    Assert.assertEquals("MyEntity", entity.name)
    
    val attribute = entity.attributes.get(0)
    Assert.assertEquals("attribute", attribute.name);
    Assert.assertEquals("MyEntity",
      (attribute.type.elementType as EntityType).
        entity.name);
  }
  ...

In this test, we parse the input and test that the AST of the parsed program has the expected structure. These tests do not add much value in the Entities DSL, but in a more complex DSL you do want to test that the structure of the parsed EMF model is as you expect (we will see an example of that in Chapter 8, An Expression Language).

You can now run the test; right-click on the Xtend file and navigate to Run As | JUnit Test. The test should pass, and you should see the green bar in the JUnit view.

Note that the parse method returns an EMF model even if the input string contains syntax errors since it tries to parse as much as it can. Thus, if you want to make sure that the input string is parsed without any syntax error, you have to check that explicitly. To do that, you can use another utility class, ValidationTestHelper. This class provides many assert methods that take an EObject argument. You can use an extension field and simply call assertNoErrors on the parsed EMF object. Alternatively, if you do not need the EMF object, but you just need to check that there are no parsing errors, you can simply call it on the result of parse, for example:

class EntitiesParsingTest {
  
  @Inject extension ParseHelper<Model>
  @Inject extension ValidationTestHelper
...
  @Test
  def void testCorrectParsing() {
    '''
      entity MyEntity {
          MyEntity attribute
      }
    '''.parse.assertNoErrors
  }

If you try to run the tests again, you will get a failure for this new test:

java.lang.AssertionError: Expected no errors, but got :
ERROR (org.eclipse.xtext.diagnostics.Diagnostic.Syntax)
'missing ';' at '}'' on Entity, offset 41, length 1

The reported error should be clear enough: we forgot to add the terminating ';' in our input program; thus, we can fix it and run the test again. This time, the green bar should be back.

You can now write other @Test methods for testing the various features of the DSL (see the sources of the examples). Depending on the complexity of your DSL, you may have to write many of them.

Remember that you should follow this methodology while implementing your DSL, not after having implemented all of it. If you follow this strictly, you will not have to launch Eclipse to manually check that you implemented a feature correctly, and you will note that this methodology will let you program really fast.

Earlier, we used the ValidationTestHelper class to test that it was possible to parse without errors. Of course, we also need to test that errors and warnings are detected. In particular, we should test any error situation handled by our own validator. The ValidationTestHelper class contains utility methods, besides assertNoErrors, that allow us to test whether the expected errors are correctly issued.

For instance, for our Entities DSL, we wrote a custom validator method that checks that the entity hierarchy is acyclic (Chapter 4, Validation). Thus, we should write a test that, given an input program with a cycle in the hierarchy, checks that such an error is indeed raised during validation.

It is better to separate JUnit test classes according to the tested features; thus, we write another JUnit class, EntitiesValidatorTest, which contains tests related to validation. The start of this new JUnit test class should look familiar:

@RunWith(XtextRunner)
@InjectWith(EntitiesInjectorProvider)
class EntitiesValidatorTest {
  
  @Inject extension ParseHelper<Model>
  @Inject extension ValidationTestHelper
  ...

We are now going to use the assertError method from ValidationTestHelper, which, besides the EMF model element to validate, requires the following arguments:

Thus, we parse input containing an entity extending itself, and we pass the arguments to assertError according to the error generated by checkNoCycleInEntityHierarchy in EntitiesValidator (see Chapter 4, Validation):

@Test
def void testEntityExtendsItself() {
  '''
    entity MyEntity extends MyEntity {

    }
  '''.parse.assertCycleInHierarchy("MyEntity")
}

def private assertCycleInHierarchy(Model m, String entityName) {
  m.assertError(
    EntitiesPackage.eINSTANCE.entity,
    EntitiesValidator.HIERARCHY_CYCLE,
    "cycle in hierarchy of entity '" + entityName + "'"
  )

}

Note that the EObject argument is the one returned by the parse method (we use assertError as an extension method). Since the error concerns an Entity object, we specify the corresponding EClass (retrieved using EntitiesPackage), the expected issue code, and finally, the expected error message. This test should pass.

We can now write another test, which tests the same validation error on a more complex input with a cycle in the hierarchy involving more than one entity. In this test, we make sure that our validator issues an error for each of the entities involved in the hierarchy cycle:

@Test
def void testCycleInEntityHierarchy() {
  '''
    entity A extends B {}
    entity B extends C {}
    entity C extends A {}
  '''.parse => [
        assertCycleInHierarchy("A")
        assertCycleInHierarchy("B")
        assertCycleInHierarchy("C")
  ]
}

You can also check that the error marker generated by the validator is created on the right element in the source file. In order to do that, you use the version of assertError that also takes the expected offset and the expected length of the text region marked with error. For example, the EntitiesValidator should generate the error for a cycle in the hierarchy on the superType feature. We write the following test to check this:

@Test
def void testCycleInHierarchyErrorPosition() {
  val testInput =
  '''
    entity MyEntity extends MyEntity {
    }
  '''
  testInput.parse.assertError(
    EntitiesPackage.eINSTANCE.entity,
    EntitiesValidator.HIERARCHY_CYCLE,
    testInput.lastIndexOf("MyEntity"), // offset
    "MyEntity".length // length
  )
}

We check that the offset and the length of the text region marked with error corresponds to the entity named after "extends", that is, the last occurrence of "MyEntity" in the input.

You can also assert warnings, using assertWarning, which has the same signatures as the assertError used in the previous code snippet. Similarly, you can use assertNoWarnings, which corresponds to assertNoErrors, but with respect to warnings. The assertIssue and assertNoIssues methods perform similar assertions without considering the severity level.

You should keep in mind that a broken implementation of a validation rule could always mark entities with errors. For this reason, you should always write a test for positive cases as well:

@Test
def void testValidHierarchy() {
  '''
    entity FirstEntity {}
    entity SecondEntity extends FirstEntity {}
  '''.parse.assertNoErrors
}

As we said in the previous chapter, the formatter is also used in a non-UI environment, thus, we can test the formatter for our DSL with plain JUnit tests. To test the formatter, we create a new Xtend class, and we inject as extension the FormatterTester class:

@RunWith(XtextRunner)
@InjectWith(EntitiesInjectorProvider)
class EntitiesFormatterTest {

    @Inject extension FormatterTester
...

To test the formatter, we use the assertFormatted method that takes a lambda where we specify the input to be formatted and the expected formatted program:

@Test
def void testEntitiesFormatter() {
    assertFormatted[
        toBeFormatted = '''
                entity E1 { int i ; string s; boolean b   ;}
                entity  E2  extends  E1{}
        '''
        expectation = '''
                ...
        '''
    ]
}

Why did we specify … as the expected formatted output? Why did we not try to specify what we really expect as the formatted output? Well, we could have written the expected output and probably we would have gotten it right on the first try, but why not simply make the test fail and see the actual output? We can then copy that in our test once we are convinced that it is correct. So let's run the test, and when it fails, the JUnit view tells us what the actual result is, as shown in the following screenshot:

If you now double-click on the line showing the comparison failure in the JUnit view, you will get a dialog showing a line-by-line comparison, as shown in the following screenshot:

You can verify that the actual output is correct, copy that, and paste it into your test as the expected output. The test will now succeed:

@Test
def void testEntitiesFormatter() {
    assertFormatted[
        toBeFormatted = '''

            entity E1 { int i ; string s; boolean b   ;}
            entity  E2  extends  E1{}
        '''
        expectation = '''
            entity E1 {
                    int i;
                    string s;
                    boolean b;
            }
            
            entity E2 extends E1 {
            }
        '''
    ]
}

Using this technique, you can easily write JUnit tests that deal with comparisons. However, remember that the Result Comparison dialog appears only if you compare String objects.

Xtext provides a helper class to test your code generator, CompilationTestHelper, which we inject as an extension field in the JUnit test class. This helper class parses an input string and runs the code generator, thus we do not need the parser helper in this test class:

@RunWith(XtextRunner)
@InjectWith(EntitiesInjectorProvider)
class EntitiesGeneratorTest {
  
  @Inject extension CompilationTestHelper

This helper class provides the method assertCompilesTo, which takes a char sequence representing an input program and a char sequence representing the expected generated output. Using that as an extension method, we can then write the following test method, which tests that the generated Java code is as we expect (we use the technique of the JUnit view to get the actual output as illustrated in the previous section):

@Test
def void testGeneratedCode() {
  '''
  entity MyEntity {
    string myAttribute;
  }
  '''.assertCompilesTo(
  '''
  package entities;

  public class MyEntity {
    private String myAttribute;

    public String getMyAttribute() {
      return myAttribute;
    }
    
    public void setMyAttribute(String _arg) {
      this.myAttribute = _arg;
    }
    
  }
  ''')
}

Testing that the generated output corresponds to what we expect is already a good testing technique. However, when the generated code is Java code, it might be good to also test that it is valid Java code, that is, the Java compiler compiles the generated code without errors.

To test that the generated code is valid Java code, we use the CompilationTestHelper.compile method, which takes an input string and a lambda. The parameter of the lambda is a Result object (an inner class). In the lambda, we can use the Result object to perform additional checks. To test the validity of the generated Java code, we can call the Result.getCompiledClass method. This method compiles the generated code with the Eclipse Java compiler. If the Java compiler issues an error, then our test will fail and the JUnit view will show the compilation errors.

We write the following test (remember that if no parameter is explicitly declared in the lambda, the special implicit variable it is used):

@Test
def void testGeneratedJavaCodeIsValid() {
  '''
  entity MyEntity {
    string myAttribute;
  }
  '''.compile[getCompiledClass]
  // check that it is valid Java code
}

If the getCompiledClass class terminates successfully, it also returns a Java Class object, which can be used to instantiate the compiled Java class by reflection. This allows us to test the generated Java class' runtime behavior. We can easily invoke the created instance's methods through the reflection support provided by the Xtext class ReflectExtensions. For example (Assert's static methods are imported statically):

@Inject extension ReflectExtensions

@Test
def void testGeneratedJavaCode() {
  '''
  entity E {
    string myAttribute;
  }
  '''.compile[
    getCompiledClass.newInstance => [
      assertNull(it.invoke("getMyAttribute"))
      it.invoke("setMyAttribute", "value")
      assertEquals("value",
        it.invoke("getMyAttribute"))
    ]
  ]
}

This method tests (through the getter method) that the attributes are initialized to null and that the setter method sets the corresponding attribute.

You can test situations when the generator generates several files originating from a single input. In the Entities DSL, a Java class is generated for each entity, thus, we can perform checks on each generated Java file by using the Result.getGeneratedCode(String) method that takes the name of the generated Java class as an argument and returns its contents. Since the generator for our Entities DSL should generate the Java class entities.E for an entity named "E", you must specify the fully qualified name to retrieve the generated code for an entity.

Similarly, we can check that the several generated Java files compile correctly. We can perform reflective Java operations on all the compiled Java classes using Result.getCompiledClass(String) specifying the fully qualified name of the generated Java class:

@Test def void testGeneratedJavaCodeWithTwoClasses() {
  '''
  entity FirstEntity {
    SecondEntity myAttribute;
  }

  entity SecondEntity {
    string s;
  }
  '''.compile[
    val FirstEntity =
      getCompiledClass("entities.FirstEntity").newInstance
    val SecondEntity =
      getCompiledClass("entities.SecondEntity").newInstance
    SecondEntity.invoke("setS", "testvalue")
    FirstEntity.invoke("setMyAttribute", SecondEntity)
    SecondEntity.assertSame(FirstEntity.invoke("getMyAttribute"))
    "testvalue".assertEquals
      (FirstEntity.invoke("getMyAttribute").invoke("getS"))
    ]
}

In particular, in this last example, the first generated Java class depends on the second generated Java class.

These tests might not be valuable in this DSL, but in more complex DSLs, having tests which automatically check the runtime behavior of the generated code enhances productivity.

The CompilationTestHelper class runs your DSL validator, but it will call the generator even if the parsed model is not valid. The output of the generator might be meaningless in such cases. If you want to make sure that's what you pass to CompilationTestHelper is a valid input for your DSL, you need to manually check whether the Result contains validation errors. This is an example of how to do that:

def void testInputWithValidationError() {
    '''
        entity MyEntity {
                // missing ;
                string myAttribute
        }
    '''.compile [
        val allErrors = getErrorsAndWarnings.filter[severity == Severity.ERROR]
        if (!allErrors.empty) {
                throw new IllegalStateException(
                        "One or more resources contained errors : " +
                        allErrors.map[toString].join(", ")
                );
        }
    ]
}

If you run this test it will fail, since the input contains a parser error.