head	1.2;
access;
symbols
	perseant-exfatfs-base-20240630:1.2
	perseant-exfatfs:1.2.0.24
	perseant-exfatfs-base:1.2
	cjep_sun2x:1.2.0.22
	cjep_sun2x-base:1.2
	cjep_staticlib_x-base1:1.2
	cjep_staticlib_x:1.2.0.20
	cjep_staticlib_x-base:1.2
	phil-wifi-20200421:1.2
	phil-wifi-20200411:1.2
	phil-wifi-20200406:1.2
	pgoyette-compat-merge-20190127:1.2
	pgoyette-compat-20190127:1.2
	pgoyette-compat-20190118:1.2
	pgoyette-compat-1226:1.2
	pgoyette-compat-1126:1.2
	pgoyette-compat-1020:1.2
	pgoyette-compat-0930:1.2
	pgoyette-compat-0906:1.2
	pgoyette-compat-0728:1.2
	pgoyette-compat-0625:1.2
	pgoyette-compat-0521:1.2
	pgoyette-compat-0502:1.2
	pgoyette-compat-0422:1.2
	pgoyette-compat-0415:1.2
	pgoyette-compat-0407:1.2
	pgoyette-compat-0330:1.2
	pgoyette-compat-0322:1.2
	pgoyette-compat-0315:1.2
	pgoyette-compat:1.2.0.18
	pgoyette-compat-base:1.2
	perseant-stdc-iso10646:1.2.0.16
	perseant-stdc-iso10646-base:1.2
	prg-localcount2-base3:1.2
	prg-localcount2-base2:1.2
	prg-localcount2-base1:1.2
	prg-localcount2:1.2.0.14
	prg-localcount2-base:1.2
	pgoyette-localcount-20170426:1.2
	bouyer-socketcan-base1:1.2
	pgoyette-localcount-20170320:1.2
	bouyer-socketcan:1.2.0.12
	bouyer-socketcan-base:1.2
	pgoyette-localcount-20170107:1.2
	pgoyette-localcount-20161104:1.2
	localcount-20160914:1.2
	pgoyette-localcount-20160806:1.2
	pgoyette-localcount-20160726:1.2
	pgoyette-localcount:1.2.0.10
	pgoyette-localcount-base:1.2
	yamt-pagecache-base9:1.2
	yamt-pagecache-tag8:1.2
	tls-earlyentropy:1.2.0.6
	tls-earlyentropy-base:1.2
	riastradh-xf86-video-intel-2-7-1-pre-2-21-15:1.2
	riastradh-drm2-base3:1.2
	agc-symver:1.2.0.8
	agc-symver-base:1.2
	tls-maxphys-base:1.2
	yamt-pagecache-base8:1.2
	yamt-pagecache-base7:1.2
	yamt-pagecache-base6:1.2
	yamt-pagecache-base5:1.2
	yamt-pagecache-base4:1.2
	yamt-pagecache-base3:1.2
	yamt-pagecache-base2:1.2
	yamt-pagecache:1.2.0.4
	yamt-pagecache-base:1.2
	bouyer-quota2-nbase:1.2
	bouyer-quota2:1.2.0.2
	bouyer-quota2-base:1.2
	atf-0-8:1.1.1.2
	atf-0-7:1.1.1.1
	TNF:1.1.1;
locks; strict;
comment	@# @;


1.2
date	2010.06.04.08.32.14;	author jmmv;	state dead;
branches;
next	1.1;

1.1
date	2009.12.22.13.25.58;	author jmmv;	state Exp;
branches
	1.1.1.1;
next	;

1.1.1.1
date	2009.12.22.13.25.58;	author jmmv;	state Exp;
branches;
next	1.1.1.2;

1.1.1.2
date	2010.05.08.08.05.23;	author jmmv;	state Exp;
branches;
next	;


desc
@@


1.2
log
@Fix import conflicts for atf 0.9.
@
text
@<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.1//EN"
                  "http://docbook.org/xml/simple/1.1/sdocbook.dtd">

<article>

<articleinfo>
  <title>Rearchitecting ATF: Road trip to 1.0</title>

  <author>
    <firstname>Julio</firstname>
    <surname>Merino</surname>
    <affiliation>
      <orgname>The NetBSD Foundation</orgname>
    </affiliation>
  </author>
</articleinfo>

<section id="overview">

<title>Overview</title>

<para>This document describes the major steps to be taken before getting
to the first stable release of ATF, 1.0.  Some of these steps describe
the transition from the old code base to the new ideas documented in the
specification (TODO: Add link somehow).</para>

</section>

<section id="plan-0-8">

<title>The plan to 0.8</title>

<itemizedlist>

<listitem><para>Add atf-sh interface to build-time tests.</para></listitem>

<listitem><para>Properly document the libraries: i.e. one page per
module, detailed information of each function and type, etc.  At the
very least atf-c.</para></listitem>

<listitem><para>Add a tool to collect multiple outputs of atf-run (from
different machines) and generate a single (XML-only?) log with
everything.  Must allow easy conversion to HTML for online
publishing.</para></listitem>

<listitem><para>Allow grouping of test programs in tiers in an Atffile.
This is to permit the user specify "dependencies" between test programs:
e.g. do not run a specific test program if its dependencies have failed,
because it will certainly fail also.</para></listitem>

<listitem><para>Provide a kernel-level unit testing API (for NetBSD
only, at the moment).  This should come in the form of an atf.ko module
that provides functions to define and register test cases, functions for
results reporting and an interface (a trivial file system?) that
transports the application/X-atf-tcs output to user-space, provides
information to user-space about the available test cases (a list) and
allows user-space to launch the execution of test
cases.</para></listitem>

</itemizedlist>

</section>

<section id="plan-0-9">

<title>The plan to 0.9</title>

<itemizedlist>

<listitem><para>Add a module to atf-c to manage dynamic memory.  Should
provide a "mem chunk" object that can only be managed through functions
(i.e. not directly) so that access to buffers can be safely controlled.
Dealing with strdup and similar functions, for example, might be
complex.</para>

<para>See these old revisions for a start, but these did not work
correctly because the use of (void **) casts brought aliasing
problems:</para>

<para>78eeacf3b9284493e5e96b7866fc1d163a13bc02
8e200683a2b70eadca728c2e0347d4790777b3fc
872393ed0177cbcc30ffacede228eb3986c42ab7</para></listitem>

</itemizedlist>

</section>

<section id="plan-pre-1-0">

<title>The plan to 1.0 prereleases</title>

<itemizedlist>

<listitem><para>Fix all occurrences of XXX, TODO and
FIXME.</para></listitem>

<listitem><para>Split the distfile into multiple components.  We should
have a component for each language binding and a component providing the
ATF tools, at the very least.  If we had this, external programs using
ATF wouldn't need to depend on the tools and/or the C++ binding, because
they could just require the user to build the atf-c
binding.</para></listitem>

<listitem><para>Think of a way to properly add tests for (almost?) all
error paths.  Most of them are probably broken
already.</para></listitem>

<listitem><para>Improve error reporting: aside from clarifying error
messages, this also implies adding more error cases to give them more
semantic meaning at the source code level..</para></listitem>

<listitem><para>Make the shell library work with 'set
-e'?</para></listitem>

<listitem><para>Shell test programs dynamically locate where the shell
library is by calling atf-config (done by atf.init.subr).  Contrarywise,
binary test programs are directly linked against the final location of
libatf.  It may be nice if the latter loaded the library dynamically
based on what atf-config said, so the user could switch atf
installations by simply changing its PATH (and effectively making atf
relocatable on the file system).  Why could this be nice?  To painlessly
run an older atf test suite against a more recent version of the code
base to ensure there are no regressions even with older tests.  Just a
crazy idea, as maybe what the shell test programs currently do is
stupid.</para></listitem>

<listitem><para>Allow users to customize the build of atf by defining
additional meta-data for test cases.  At the moment this is possible
because the meta-data is not sanity-checked, but I think it should be.
Following the previous item, NetBSD could add a 'netbsd.pr' variable and
then use this data when generating reports to add direct links to the
appropriate PRs.</para></listitem>

<listitem><para>Make sure that the tests in tests/atf have, at the very
least, the same coverage as the ones in
tests/bootstrap.</para></listitem>

<listitem><para>Document the code.</para></listitem>

<listitem><para>Possibly add a way to automatically gain or drop
privileges when require.user is set.</para></listitem>

<listitem><para>Add a way to specify which bug/issue/whatever a given
test case is stress-testing.  This information is useful when detecting
regressions.</para></listitem>

</itemizedlist>

</section>

<section id="plan-1.0-rc">

<title>The plan to 1.0 release candidates</title>

<itemizedlist>

<listitem><para>Build libatf as a shared library and set -version-info
accordingly.</para></listitem>

<listitem><para>Set the DTDs' versions to 1.0.</para></listitem>

</itemizedlist>

</section>

<section id="plan-unknown">

<title>The plan to releases after 1.0</title>

<itemizedlist>

<listitem><para>Allow the parallel execution of tests.  Achieving this
with a test program granularity is easy: only need to change atf-run.
Lowering it to a finer granularity (test cases) is harder and maybe not
worth it.</para></listitem>

</itemizedlist>

</section>

</article>
@


1.1
log
@Initial revision
@
text
@@


1.1.1.1
log
@Import atf 0.7.  Changes in this release:

* Added build-time checks to atf-c and atf-c++.  A binding for atf-sh
  will come later.

* Migrated all build-time checks for header files to proper ATF tests.
  This demonstrates the use of the new feature described above.

* Added an internal API for child process management.

* Converted all plain-text distribution documents to a Docbook canonical
  version, and include pre-generated plain text and HTML copies in the
  distribution file.

* Simplified the contents of the Makefile.am by regenerating it from a
  canonical Makefile.am.m4 source.  As a side-effect, some dependency
  specifications were fixed.

* Migrated all checks from the check target to installcheck, as these
  require ATF to be installed.

* Fixed sign comparison mismatches triggered by the now-enabled
  -Wsign-compare.

* Fixed many memory and object leaks.
@
text
@@


1.1.1.2
log
@Import atf 0.8.  Changes in this release:

* Test programs no longer run several test cases in a row. The execution
  of a test program now requires a test case name, and that single test
  case is executed. To execute several test cases, use the atf-run
  utility as usual.

* Test programs no longer fork a subprocess to isolate the execution of
  test cases. They run the test case code in-process, and a crash of the
  test case will result in a crash of the test program. This is to ease
  debugging of faulty test cases.

* Test programs no longer isolate their test cases. This means that they
  will not create temporary directories nor sanitize the environment any
  more. Yes: running a test case that depends on system state by hand
  will most likely yield different results depending on where (machine,
  directory, user environment, etc.) it is run. Isolation has been moved
  to atf-run.

* Test programs no longer print a cryptic format (application/X-atf-tcs)
  on a special file channel. They can now print whatever they want on
  the screen. Because test programs can now only run one test case every
  time, providing controlled output is not necessary any more.

* Test programs no longer write their status into a special file
  descriptor. Instead, they create a file with the results, which is
  later parsed by atf-run. This changes the semantics of the -r flag.

* atf-run has been adjusted to perform the test case isolation. As a
  result, there is now a single canonical place that implements the
  isolation of test caes. In previous releases, the three language
  bindings (C, C++ and shell) had to be kept in sync with each other
  (read: not a nice thing to do at all). As a side effect of this
  change, writing bindings for other languages will be much, much easier
  from now on.

* atf-run forks test programs on a test case basis, instead of on a test
  program basis as it did before. This is to provide the test case
  isolation that was before implemented by the test programs themselves.

* Removed the atf-exec tool. This was used to implement test case
  isolation in atf-sh, but it is now unnecessary.

* It is now optional to define the descr meta-data property. It has been
  proven to be mostly useless, because test cases often carry a
  descriptive name of their own.
@
text
@d30 1
a30 1
<section id="plan-0-9">
d32 1
a32 1
<title>The plan to 0.9</title>
d65 1
a65 1
<section id="plan-0-10">
d67 1
a67 1
<title>The plan to 0.10</title>
a88 88
<section id="plan-misc">

<title>The plan to fix miscellaneous issues</title>

<itemizedlist>

<listitem><para>Consider getting rid of the checked-in top-level documents.
Don't know why we are doing that at all.</para></listitem>

<listitem><para>Get rid of the "C object model", e.g.
<literal>atf_object_t</literal>.  It is completely useless at this point
and just causes pain.  If we want to catch leaks, let's just use Valgrind
in a platform where this tool works.</para></listitem>

<listitem><para>Validate test program command line syntax: -r has to be
invalid when running the cleanup part of a test case.</para></listitem>

<listitem><para>Make test case list sort properties alphabetically?  Wrap
them on line boundaries when printing to terminal?</para></listitem>

<listitem><para>Proper cleanup of a test case when ctrl+c from
atf-run.</para></listitem>

<listitem><para>When running tests/atf/atf-run/t_atf_run manually, it
overwrites Atffile.  This is a bit annoying and should be somehow
fixed.</para></listitem>

<listitem><para>Templatize the void* cookie in atf-c++'s
fork.</para></listitem>

<listitem><para>Remove obsolete helper tests in tests/atf/test_programs/h_*
for cleanup.</para></listitem>

<listitem><para>Document the applications/X-atf-tp
format.</para></listitem>

<listitem><para>Update atf-test-program(4) to reflect the command-line
changes.</para></listitem>

<listitem><para>Update atf-test-case(4) to reflect the isolation
changes.</para></listitem>

<listitem><para>Fix default for -r flag in test programs.</para></listitem>

<listitem><para>Fix any XXX and TODO items introduced by this
branch.</para></listitem>

<listitem><para>Get rid of atf-compile.  Instead, just write a wrapper
around sh that loads the necessary library files and then executes the test
program.  Make it be usable with #! /...</para></listitem>

<listitem><para>Drop the tests/ hierarchy and intermix test programs with
the source files.  (The installed layout would remain the same.)  Rename
t_* to *_test.</para></listitem>

<listitem><para>Get rid of the data/ subdirectory: just put the files where
they belong.</para></listitem>

<listitem><para>Get rid of h_{pass,fail,skip} duplicates.</para></listitem>

<listitem><para>Add a use.fs metadata property, defaulting to
no.</para></listitem>

<listitem><para>Reverse arguments to ATF_CHECK_THROW.  Also add a macro to
check the error message of an exception, alongside its
type.</para></listitem>

<listitem><para>Get rid of the atf::tests::vars_map monstrosity and prune
unnecessary inclusions of atf-c++/tests.hpp.  I'm thinking that we need a
simple "types" file containing convenience types used thorough the code,
and that we need a "strong_typedef" wrapper for them.</para></listitem>

<listitem><para>Do not use std::endl.  Understand what it is for first, and
then decide.</para></listitem>

<listitem><para>It looks like atf-c++/signals may be unused.  Review and
delete as required.</para></listitem>

<listitem><para>The suggested <literal>atf-run | atf-report</literal> does
not work quite well because most shells lose the exit code of the first
command in the pipeline.  Therefore, it's not possible to get the result
of the execution of the test suite.  Find a way around this; maybe implement
a wrapping tool.</para></listitem>

</itemizedlist>

</section>

@