First login?
Lost password?
 
 
cellml-tools-developers AT lists.cellml.org 
A list for the developers of CellML tools

Text archives Help


[cellml-dev] Documentation and Litmus tests


Chronological Thread 
  • From: r.britten at auckland.ac.nz (Randall Britten)
  • Subject: [cellml-dev] Documentation and Litmus tests
  • Date: Mon, 12 Nov 2007 10:35:45 +1300
Hi all

I think Andrew has made a very good point regarding why black-box Litmus
tests are preferable. It is different from the way I usually thought about
this, but I like it, and accept that we should use it.

Regarding documentation, it seems to me that we still need a way of ensuring
we don't forget to document new or changed features. I don't have any
particular preference for how we do this, but I do think we need to have
some part of our process that ensures this.

Regards,
Randall

P.S., I would just like to clarify that I am no fan of bureaucracy and
burdensome processes, let's keep them as light as possible.

> -----Original Message-----
> From: cellml-tools-developers-bounces at cellml.org [mailto:cellml-tools-
> developers-bounces at cellml.org] On Behalf Of Andrew Miller
> Sent: Friday, 9 November 2007 6:00 p.m.
> To: A list for the developers of CellML tools
> Subject: Re: [cellml-dev] Documentation and Litmus tests
>
> Randall Britten wrote:
> > My original question leaves open the option of batching docs and
> litmus
> > tests.
> >
> > So, we still need to formalise how we ensure that required updates to
> docs
> > and Litmus tests are not forgotten.
>
> I don't think there is a 1-1 mapping between documentation and tracker
> items, or tracker items and Litmus tests. By batching the tests, we are
> re-auditing the list of tests / the available documentation every time
> to ensure that they correspond to the current features visible to the
> end-user. The way we currently use to do this is by enumerating through
> every possible distinct UI option that the user has - for example, the
> user has an option of which top-level menu to choose. We then
> recursively repeat this process on each sub-menu option, and likewise
> for every other control available in PCEnv. The person updating the
> functional tests is expected to know how to test that a control has
> worked appropriately, but there is usually only one main path to test
> that.
>
> In general, functional tests should be as 'black-box' as possible, and
> therefore driven from the functional end as I described and not the
> code
> end. Many tracker items are likely to come from the functional tests
> (or
> something already covered by them).
>
> In some cases there can be more complex issues that involve special
> combinations of performing unusual combinations of actions on different
> controls. I am not aware of any tracker items which occur in this
> category, and if there are any, they would certainly be too rare to
> justify taking some action for every tracker item - instead, perhaps
> such tracker items could have a keyword set on them. Unless you are
> aware of a significant number of tracker items needing this, however, I
> think we should strive to avoid introducing too much bureaucracy into
> the process - the time costs of following each process adds up quite
> quickly, and if a certain process has only hypothetical benefits, we
> are
> likely to be better off without it.
>
> Best regards,
> Andrew
>
> >
> > Regards,
> > Randall
> >
> >> -----Original Message-----
> >> From: cellml-tools-developers-bounces at cellml.org [mailto:cellml-
> tools-
> >> developers-bounces at cellml.org] On Behalf Of Andrew Miller
> >> Sent: Friday, 9 November 2007 2:44 p.m.
> >> To: A list for the developers of CellML tools
> >> Subject: Re: [cellml-dev] Documentation and Litmus tests
> >>
> >> Randall Britten wrote:
> >>> Hi all
> >>>
> >>>
> >>>
> >>> Any preferences as to how we can update our development process so
> >>> that any time we add a new feature, we have a process for ensuring
> >>> that the documentation and Litmus tests for these get added?
> >>>
> >>>
> >>>
> >>> Some possible options: using flags on the tracker items; or not
> >>> resolving them when the code is done, but rather assigning them to
> >> the
> >>> person responsible for writing the documentation?
> >>>
> >> Hi,
> >>
> >> I personally think we should be able to mark bugs which are fixed as
> >> being fixed even if we don't yet have tests, otherwise it is a bit
> >> misleading (and also, it makes it harder to use QA states like
> VERIFIED
> >> should we want to do that).
> >>
> >> I think the first question we should ask is if we want to do batched
> or
> >> continuous updates to functional tests and documentation. The
> batched
> >> mode we are currently using does seem to work well as far as I'm
> aware:
> >> a) Documentation-wise, the documentation should provide both a
> >> reference and a guide through the program. Guide-type documentation
> >> really does need to be batched, because it is highly dependent on
> how
> >> features interact, and we don't want to keep re-writing large chunks
> of
> >> it continuously. Reference type documentation may be easier to write
> >> continuously so the developer has fresh memory of all the details,
> >> although there is a risk of focusing too much on the technical
> >> specifics
> >> and not enough on how it would actually be used.
> >>
> >> b) Functional tests are often like a more comprehensive guide
> through
> >> the program - it is sometimes easier to check that we have all
> >> functional tests by enumerating the paths through the UI that you
> get
> >> at
> >> them, and this is really what functional tests are supposed to be.
> >> Regressions would normally trigger a failure of a functional test
> >> created this way if the full functional tests are comprehensive.
> >>
> >> Of course, automated unit / regression tests are a different story
> >> entirely - because they are run so much more often, and test lower-
> >> level
> >> rather than functional elements, they should be updated for bugs (we
> >> really need to get a framework for this set up in PCEnv). In this
> case,
> >> keywords or perhaps flags may be appropriate. We could use flags in
> the
> >> sense that:
> >> 1) Someone sees something that they think should have the test-
> case,
> >> so they request the flag.
> >> 2) The flag sites there waiting for either a test to be written
> and
> >> the flag +'d, or for consensus that a test wouldn't be appropriate
> to
> >> be
> >> reached and the flag minused.
> >>
> >> Keywords would be another option if we want tracker items to require
> a
> >> test by default - then we could search for those that are resolved
> >> without the keyword - we would probably need another keyword to say
> >> that
> >> it wouldn't make sense to write a test for a given tracker item to
> make
> >> this usable.
> >>
> >> Best regards,
> >> Andrew
> >>
> >>>
> >>> Regards,
> >>>
> >>> Randall
> >>>
> >>> -------------------------------------------------------------------
> --
> >> ---
> >>> _______________________________________________
> >>> cellml-tools-developers mailing list
> >>> cellml-tools-developers at cellml.org
> >>> http://www.cellml.org/mailman/listinfo/cellml-tools-developers
> >>>
> >> _______________________________________________
> >> cellml-tools-developers mailing list
> >> cellml-tools-developers at cellml.org
> >> http://www.cellml.org/mailman/listinfo/cellml-tools-developers
> >
> > _______________________________________________
> > cellml-tools-developers mailing list
> > cellml-tools-developers at cellml.org
> > http://www.cellml.org/mailman/listinfo/cellml-tools-developers
>
> _______________________________________________
> cellml-tools-developers mailing list
> cellml-tools-developers at cellml.org
> http://www.cellml.org/mailman/listinfo/cellml-tools-developers



Archive powered by MHonArc 2.6.18.

Top of page