Rearchitecting ATF: Road trip to 1.0
This document describes the major steps to be taken before getting to the first stable release of ATF, 1.0. Some of these steps describe the transition from the old code base to the new ideas documented in the specification (TODO: Add link somehow).
Add atf-sh interface to build-time tests.
Properly document the libraries: i.e. one page per module, detailed information of each function and type, etc. At the very least atf-c.
Add a tool to collect multiple outputs of atf-run (from different machines) and generate a single (XML-only?) log with everything. Must allow easy conversion to HTML for online publishing.
Allow grouping of test programs in tiers in an Atffile. This is to permit the user specify "dependencies" between test programs: e.g. do not run a specific test program if its dependencies have failed, because it will certainly fail also.
Provide a kernel-level unit testing API (for NetBSD only, at the moment). This should come in the form of an atf.ko module that provides functions to define and register test cases, functions for results reporting and an interface (a trivial file system?) that transports the application/X-atf-tcs output to user-space, provides information to user-space about the available test cases (a list) and allows user-space to launch the execution of test cases.
Add a module to atf-c to manage dynamic memory. Should provide a "mem chunk" object that can only be managed through functions (i.e. not directly) so that access to buffers can be safely controlled. Dealing with strdup and similar functions, for example, might be complex.
See these old revisions for a start, but these did not work correctly because the use of (void **) casts brought aliasing problems:
78eeacf3b9284493e5e96b7866fc1d163a13bc02 8e200683a2b70eadca728c2e0347d4790777b3fc 872393ed0177cbcc30ffacede228eb3986c42ab7
Fix all occurrences of XXX, TODO and FIXME.
Split the distfile into multiple components. We should have a component for each language binding and a component providing the ATF tools, at the very least. If we had this, external programs using ATF wouldn't need to depend on the tools and/or the C++ binding, because they could just require the user to build the atf-c binding.
Think of a way to properly add tests for (almost?) all error paths. Most of them are probably broken already.
Improve error reporting: aside from clarifying error messages, this also implies adding more error cases to give them more semantic meaning at the source code level..
Make the shell library work with 'set -e'?
Shell test programs dynamically locate where the shell library is by calling atf-config (done by atf.init.subr). Contrarywise, binary test programs are directly linked against the final location of libatf. It may be nice if the latter loaded the library dynamically based on what atf-config said, so the user could switch atf installations by simply changing its PATH (and effectively making atf relocatable on the file system). Why could this be nice? To painlessly run an older atf test suite against a more recent version of the code base to ensure there are no regressions even with older tests. Just a crazy idea, as maybe what the shell test programs currently do is stupid.
Allow users to customize the build of atf by defining additional meta-data for test cases. At the moment this is possible because the meta-data is not sanity-checked, but I think it should be. Following the previous item, NetBSD could add a 'netbsd.pr' variable and then use this data when generating reports to add direct links to the appropriate PRs.
Make sure that the tests in tests/atf have, at the very least, the same coverage as the ones in tests/bootstrap.
Document the code.
Possibly add a way to automatically gain or drop privileges when require.user is set.
Add a way to specify which bug/issue/whatever a given test case is stress-testing. This information is useful when detecting regressions.
Build libatf as a shared library and set -version-info accordingly.
Set the DTDs' versions to 1.0.
Allow the parallel execution of tests. Achieving this with a test program granularity is easy: only need to change atf-run. Lowering it to a finer granularity (test cases) is harder and maybe not worth it.
The plan to fix miscellaneous issues d65 1 a65 1 The plan to 0.9 d116 1 a116 1 The plan to 0.10 a141 161 The plan to fix miscellaneous issues
Consider getting rid of the checked-in top-level documents. Don't know why we are doing that at all.
Get rid of the "C object model", e.g. atf_object_t. It is completely useless at this point and just causes pain. If we want to catch leaks, let's just use Valgrind in a platform where this tool works.
Validate test program command line syntax: -r has to be invalid when running the cleanup part of a test case.
Make test case list sort properties alphabetically? Wrap them on line boundaries when printing to terminal?
Proper cleanup of a test case when ctrl+c from atf-run.
When running tests/atf/atf-run/t_atf_run manually, it overwrites Atffile. This is a bit annoying and should be somehow fixed.
Templatize the void* cookie in atf-c++'s fork.
Remove obsolete helper tests in tests/atf/test_programs/h_* for cleanup.
Document the applications/X-atf-tp format.
Update atf-test-program(4) to reflect the command-line changes.
Update atf-test-case(4) to reflect the isolation changes.
Fix default for -r flag in test programs.
Fix any XXX and TODO items introduced by this branch.
Get rid of atf-compile. Instead, just write a wrapper around sh that loads the necessary library files and then executes the test program. Make it be usable with #! /...
Drop the tests/ hierarchy and intermix test programs with the source files. (The installed layout would remain the same.) Rename t_* to *_test.
Get rid of the data/ subdirectory: just put the files where they belong.
Get rid of h_{pass,fail,skip} duplicates.
Add a use.fs metadata property, defaulting to no.
Reverse arguments to ATF_CHECK_THROW. Also add a macro to check the error message of an exception, alongside its type.
Get rid of the atf::tests::vars_map monstrosity and prune unnecessary inclusions of atf-c++/tests.hpp. I'm thinking that we need a simple "types" file containing convenience types used thorough the code, and that we need a "strong_typedef" wrapper for them.
Do not use std::endl. Understand what it is for first, and then decide.
It looks like atf-c++/signals may be unused. Review and delete as required.
The suggested atf-run | atf-report does not work quite well because most shells lose the exit code of the first command in the pipeline. Therefore, it's not possible to get the result of the execution of the test suite. Find a way around this; maybe implement a wrapping tool.