Before getting started with your static analysis definition, you set up a test suite for static analysis. Develop the test suite in tandem with the development of your static analysis. The test suite consists of positive and negative test cases. We will not grade this test suite, but you should develop one to get confidence in the quality of your static analysis. When you are asking for help from the course staff, we will first ask what tests you have written to demonstrate the problem.
In your chocopy project, develop a test suite for static analysis. The test suite should provide
See the example tests in the WebLab homework assignments for Week 3.
In test cases for reference resolution,
you write syntactically correct programs and
mark names at definition and use sites with inner square bracket blocks.
You can then relate the use site with the definition site in a resolve x to y clause,
using numbers to refer to the inner blocks.
For example, the following two test cases require to resolve the type Foo to the name in the definition of class Foo:
module resolution
test class name resolution[[
class [[A]](object):
pass
class B(object):
a:[[A]] = None
]] resolve #2 to #1
test class field resolution[[
class Foo(object):
[[bar]]:int = 0
foo:Foo = None
foo = Foo()
print(foo.[[bar]])
]] resolve #2 to #1
After copying this into an SPT file Spoofax will add the error “Reference resolution failed” and “No constraint generation rule for …“. This is expected, since your project is missing an implementation for reference resolution (this is part of the next lab).
You can use fixtures to avoid repeating parts in similar test cases. See the SPT documentation for details.
You should come up with test cases for the resolution of class names, field names, parameter names, and variable names. Start with simple test cases, but keep in mind that coverage is the main criterion for your grade. It is important to think about forward and backward references, global and non-local declarations in functions, and resolution in the presence of homonyms.
Make sure that there are no errors in tests with a resolve x to y clause, these tests are invalid when there are errors.
In test cases for error checking, you need to specify the number of errors, warnings, or notes in a
test case in errors, warnings, or notes clauses. For example, the following test cases
specify a correct ChocoPy program, a program with two errors which are reported on the name of a
duplicate class Foo, and another program with an error which is reported on the name of an unknown
class Bar:
module correctness
language chocopy
test correct program [[
class Counter(object):
count:int = 0
def getCount(self:Counter)->int:
return self.count
def increaseCount(self:Counter):
self.count = self.count + 1
]] 0 errors
test incorrect program [[
class Counter(object):
count:bool = False
def getCount(self:Counter)->int:
return self.count
def increaseCount(self:Counter):
self.count = self.count + 1
]] >= 1 errors // or 3 errors
test error on unknown class [[
class Foo(object):
bar:Bar = None
]] >= 1 errors
You can start with test cases for duplicate and missing definitions. Similar to your syntax test
cases, you can pair up positive (0 errors) and negative test cases. For duplicate definitions, we
expect errors on the definitions with the same name.
The number of errors can be hard to predict, because errors sometimes cascade. Therefore, if you
expect any errors, you should use the >= 1 errors expectation, even if you expect a specific
number of errors. For example, this expectation was used in the duplicate class test, even though we
would expect exactly two errors.
Next, you should develop test cases for fields and variables which hide global variables, global and non-local declarations, and class instantiation, subclassing, referencing. Again, you should keep in mind that coverage is the main criterion for your grade.
In test cases for type analysis,
you write syntactically correct programs and
mark expressions with inner square bracket blocks.
You can then specify the expected type of the marked expression in a run x to y clause.
For example, the following two test cases require an integer literal to be of type Int()
and a variable reference to be of its declared type Bool():
module types
test integer literal [[
print([[1]])
]] run get-type on #1 to Int()
test boolean condition [[
b1:bool = True
b2:bool = False
if [[b1 and b2]]:
print("Yes!")
]] run get-type on #1 to Bool()
In order for these tests to succeed, you need to define your own Stratego rule get-type. For this, you can copy and paste the following code into src/main.str2
signature
sorts
Type
constructors
Int : Type
Bool : Type
String : Type
ClassType : scope * ID -> Type
List : Type -> Type
NoneType : Type
EmptyList : Type
Object : Type
FunType : Type * list(Type) -> Type
// We expect to get these types in the grading pipeline
// you are free to use your own (custom) types in your Statix file
// as long as you write transformation rules from your types to our types
// in resolve-type
// Here you can define the signature of the types you have used in your Statix definitions.
// This is just an example on how to do it.
// signature
// sorts
// MyCustomType
//
// constructors
// MyCustomIntType : MyCustomType
// MyCustomClassType : ID * scope -> MyCustomType
rules
// get-type: SimpleStatement(expr) -> <get-type> expr // An example on how to match on AST nodes.
get-type: expr -> type' // Defines a rule which matches on node and returns type'
where
// Assigns variable a to be the result of the Statix analysis of the entire program (or throws an error)
a := <stx-get-ast-analysis <+ fail-msg(|$[no analysis on node [<strip-annos;write-to-string> expr]])>;
// Gets the type of the given node (or throws an error)
type := <stx-get-ast-type(|a) <+ fail-msg(|$[no type on node [<strip-annos;write-to-string> expr]])> expr;
// Calls a rule to convert the type given by Statix to a type for our tests.
type' := <resolve-type> type
// resolve-type: MyCustomIntType() -> Int() // This rule matches on your custom type, and returns a type we need in our tests.
// resolve-type: MyCustomClassType(name, scope) -> ClassType(scope, name) // This rule matches on your custom type, and returns a type we need in our tests.
// TODO: Define your own transformation rules to transform types from your Statix defintion to the types we expect in our tests
resolve-type: a -> a // This rule matches on a and returns a (trivial rule given as example). This implies no transformation actually takes place. Define your own rules ABOVE this rule.
fail-msg(|msg) = err-msg(|$[get-type: [msg]]); fail
This code defines a Stratego rule get-type which we can use to map AST Nodes to their types using the results of the Statix analysis. In our tests for the Early Feedback and Grading, we expect certain types, so it is highly recommended to use the same types in your Statix rules. If not, you might need to perform some extra transformations from your types to our types in the resolve-type rule.
For more information on the Stratego language, visit the Stratego documentation.
You can use fixtures to avoid repeating parts in similar test cases. See the SPT documentation for details.
When applying get-type to objects, we expect a ClassType constructor.
test class type [[
class Foo(object):
pass
[[Foo()]]
]] run get-type on #1 to ClassType(_, "Foo") // We don't care about the scope, only the classname.
You should come up with test cases for the types of all kinds of expressions. Just like previous testing assignments, this assignment is all about the coverage of your test suite.
The constructors for various types are:
Int()Bool()String()List(T)EmptyList()NoneType()ClassType(s, "Foo")FunType(RT, PTs)Object()Tests that contain a run x to y clause are invalid if the semantic analysis yields errors. Ensure that the snippet of ChocoPy that you are testing does not contain any errors if you want to use get-type or other strategies.
If you use a start symbol other than Program, replace any samples in this guide that use Program with your start symbol instead.
Make sure to actually append your AST nodes with their types using @x.type := T (in a similar way to assigning the ref property, which is needed for name resolution). Otherwise the rule will not find any types on your AST node.
Consider the following test case as an example:
test method name resolution [[
class Foo(object):
def [[run]](self:Foo)->int:
return 1
Foo().[[run]]()
]] resolve #2 to #1
The type of the callee expression determines the class in which the method declaration can be found.
In this example, the expression Foo() is of type ClassType(_, "Foo") and
the corresponding class Foo contains a method declaration for run().
You should come up with test cases for the resolution of method names. Start with simple test cases, but keep in mind that method name resolution is quite complex and that coverage is the main criterion for your grade. It is important to think about forward and backward references, resolution in the presence of homonyms and overriding, and the influence of class hierarchies on resolution.
You should also come up with test cases for error checking on method names.
This should include test cases for errors on duplicate definitions, missing definitions, and method overloading.
Similar to previous test cases, you can pair up positive (0 errors) and negative test cases.
Make sure that there are no errors in tests with a resolve x to y clause. These tests are invalid when there are errors.
A type error occurs, when the type of an expression does not conform to its expected type. Consider the following test case as an example:
test print boolean [[
def printInt(i:int):
print(i)
printInt(True)
]] 1 error
Type errors can occur in statements, expressions, and method declarations. You should come up with test cases for such errors. Subtyping is a common source for errors not only in programs, but also in language implementations. It is therefore important to have positive and negative typing tests, which involve correct and incorrect subtyping.
Again, keep in mind that coverage is the main criterion for your grade.
Similar to the previous testing lab, you need to be careful about the number of errors, because
errors sometimes cascade. For example, if you expect 2 errors, you should use the >= 2 errors
expectation, even if you expect an exact number of errors.
A great way of testing expected ChocoPy behavior, when you want to know whether something is allowed or not, is to try it out on the ChocoPy Website