Testing Kate ============== Author: Leo Savernik Kate contains regression tests to ensure that fixed bugs do not reappear in newer versions. To facilitate regression testing, a dedicated application testkateregression will execute the regression tests and compare them to the expecting results, indicating passed as well as failed testcases. 1. Using testkateregression -------------------------- We tried to make regression testing for Kate as easy as possible such that you can run it before each commit and find out regressions caused by your changes before they are shipped as part of a release. Running all regression tests works by simply invoking > make check in your kate build directory. While executing, testkateregression prints a line for each executed testcase, prefixed with "PASS" if it passed, and "FAIL" if it failed. Furthermore, testkateregression stores a comprehensive output log under <katetests-directory>/output/index.html. The output log is invaluable for determining why a certain testcase failed. If you invoke testkateregression the first time, it will print instructions on how to fetch the testsuite and pointing testkateregression to it. This setup has only to be done once per branch. 2. Discriminating your regressions against existing regressions -------------------------------------------------------------- In an ideal universe, all testcases always pass. In this universe, however, some testcases fail, be it because of anticipating future features not implemented yet, be it because of nasty bugs which cannot be repaired easily. This means if you've hacked on kate for quite some while and then fire up "make check", you are likely to see many failed tests pass by, most of them *not* caused by your very changes, as they failed already before. To discriminate the failed tests caused by your changes against the unaffected failures, testkateregression provides the option --save-failures=<name>, which runs the regression tests and stores all failures under a failure snapshot identified by <name>. The next time you run "make check", testkateregression automatically picks up the most recently stored failure snapshot and compares the failures and passes with the one stored in the snapshot. Each failure not listed in the failure snapshot will be prefixed with "FAIL (new)", indicating that this is a new failure. Testcases which failed in the snapshot but do pass now are prefixed with "PASS (new)", indicating that this testcase seems to be fixed now. 3. Using testkateregression efficiently -------------------------------------- Therefore, to get the most out of regression testing, we suggest the following development approach: 1. Before you change Kate, update and run testkateregression in the part- subdirectory. > make testkateregression && ./testkateregression --save-failures=last This will produce a failure snapshot called "last". 2. Hack on Kate. 3. Before you commit, run > make check It will automatically pick up the failure snapshot "last" (provided you didn't generate a newer one in the meantime) and compare all results with the previously stored ones. If you inspect <katetests-directory>/output/index.html, the new failures are marked red. Those are of interest to you, because they have been caused by your changes. New passes are marked green. These were former failures which started working due to your changes. Goto 2 while there are any new failures. 4. Commit. 4. Invoking testkateregression directly -------------------------------------- While make check is handy and simple enough for the common case, you might sometimes need more control over regression testing. testkateregression features a broad range of options, enabling you to run dedicated testcases only, specifying an alternate output directory for the logs, etc. > ./testregression --help will provide you with a complete list of options. 5. Structure of the regression test suite ---------------------------------------- Kate's regression testsuite is located in the KDE repository under trunk/tests/katetests/regression and consists of two subdirectories baseline tests The latter, tests, contains a directory hierarchy for all testcases to be run by testkateregression. The former, baseline, contains results as they are expected by correct operation. Mismatch between the output of a test and its baseline is considered to be a failure. Each directory under tests may optionally contain one of the following files. .kateconfig .kateconfig-commands ignore KNOWN_FAILURES .kateconfig: This file works exactly like .kateconfig as supported by the kate and kwrite editors. It may contain any kate line variable necessary to set up the testcases proper. Note that .kateconfig files from parent directories are not merged with .kateconfig files from child directories. .kateconfig-commands: This file may contain all commands that can be entered by kate's command line (F7). Each line will be interpreted as one command. To the contrary of .kateconfig, .kateconfig-commands files are merged with .kateconfig-commands files from parent directories. Nearer ancestors' commands take precedence over farther ancestors'. ignore: This file specifies on each line a file to be ignored in the directory the ignore-file is located. This enables you to mark any helper files which otherwise would be interpreted as testcases. Note that hidden files (.*) are ignored by default, and cannot be "unignored". KNOWN_FAILURES: This file specifies on each line a file name of a testcase which is known to fail. Such known failures are counted towards the total count of failures but they don't cause testkateregression to return a failure code. 6. Structure of a testcase ------------------------- A testcase is comprised of a simple plain text file <testcase>.txt which may be located in any subdirectory under tests. This file contains the *initial* content the testcase operates on. Each <testcase>.txt must be accompanied with a <testcase>.txt-script which contains the actual tests to be performed on the testcase. It consists of simple JavaScript-statements for direct interfacing with Kate. Last but not least, a <testcase>.txt-result exists under the baseline subdirectory, which contains a mirrored directory hierarchy of tests. This very file contains the expected *result* of the performed tests. 7. Writing a simple testcase --------------------------- Writing your own testcases is easy once you know how to get started. Let's test how Kate's C-Style indenter fares with indenting after opening braces. First, we create the new initial content under tests/indent/csmart/openbrc.txt and fill it with (the dashed lines are not part of the content) --------------------------- int main() { --------------------------- Now, we need to write a script performing some actions. We therefore create a file tests/indent/csmart/openbrc.txt-script and fill it with --------------------------- v.setCursorPosition(1,12); v.enter(); v.type("good"); --------------------------- Here, we set the initial cursor position to line 2 (the coordinates are zero- based) and column 13 which happens to be just after the opening brace. Then v.enter() simulates pressing the return key in the editor, thus inserting a new line. v.type simulates typing of the word "good" at the current position of the cursor. The options under .kateconfig specify the C-Style indenter to be applied to the testcases and an indent width of two. With this information, we know what we expect as a result. What we are still missing is the expected result itself which we create under baseline/indent/csmart/openbrc.txt-result and fill it with --------------------------- int main() { good --------------------------- You can see that "good" is indented by two spaces, even though we didn't specify those with v.type. We expect from the indenter to provide them for us. Last but not least we test the testcase by invoking in kate's part directory > ./testkateregression indent/csmart/openbrc.txt and checking whether it works the way we intended it. 7. The JavaScript-interface to the testcases ------------------------------------------- testkateregression provides you with the following global objects for each testcase: v - object of view d - object of document Each object provides the same methods and fields as the respective JavaScript- interfaces built in to Kate, like v.setCursorPosition. Additionally, v provides the following methods unique to testkateregression. type(<string>) Inserts <string> into the current cursor position as if <string> had been typed by the keyboard. Contrary to insert(<string>), it will trigger indentation and other checks. enter(), returnKey() Inserts a new line as if the return key had been pressed. This will trigger special indentation rules.