First login?
Lost password?
 
 
cellml-tools-developers AT lists.cellml.org 
A list for the developers of CellML tools

Text archives Help


[cellml-dev] Documentation and Litmus tests


Chronological Thread 
  • From: ak.miller at auckland.ac.nz (Andrew Miller)
  • Subject: [cellml-dev] Documentation and Litmus tests
  • Date: Fri, 09 Nov 2007 18:00:13 +1300
Randall Britten wrote:
> My original question leaves open the option of batching docs and litmus
> tests.
>
> So, we still need to formalise how we ensure that required updates to docs
> and Litmus tests are not forgotten.

I don't think there is a 1-1 mapping between documentation and tracker
items, or tracker items and Litmus tests. By batching the tests, we are
re-auditing the list of tests / the available documentation every time
to ensure that they correspond to the current features visible to the
end-user. The way we currently use to do this is by enumerating through
every possible distinct UI option that the user has - for example, the
user has an option of which top-level menu to choose. We then
recursively repeat this process on each sub-menu option, and likewise
for every other control available in PCEnv. The person updating the
functional tests is expected to know how to test that a control has
worked appropriately, but there is usually only one main path to test that.

In general, functional tests should be as 'black-box' as possible, and
therefore driven from the functional end as I described and not the code
end. Many tracker items are likely to come from the functional tests (or
something already covered by them).

In some cases there can be more complex issues that involve special
combinations of performing unusual combinations of actions on different
controls. I am not aware of any tracker items which occur in this
category, and if there are any, they would certainly be too rare to
justify taking some action for every tracker item - instead, perhaps
such tracker items could have a keyword set on them. Unless you are
aware of a significant number of tracker items needing this, however, I
think we should strive to avoid introducing too much bureaucracy into
the process - the time costs of following each process adds up quite
quickly, and if a certain process has only hypothetical benefits, we are
likely to be better off without it.

Best regards,
Andrew

>
> Regards,
> Randall
>
>> -----Original Message-----
>> From: cellml-tools-developers-bounces at cellml.org [mailto:cellml-tools-
>> developers-bounces at cellml.org] On Behalf Of Andrew Miller
>> Sent: Friday, 9 November 2007 2:44 p.m.
>> To: A list for the developers of CellML tools
>> Subject: Re: [cellml-dev] Documentation and Litmus tests
>>
>> Randall Britten wrote:
>>> Hi all
>>>
>>>
>>>
>>> Any preferences as to how we can update our development process so
>>> that any time we add a new feature, we have a process for ensuring
>>> that the documentation and Litmus tests for these get added?
>>>
>>>
>>>
>>> Some possible options: using flags on the tracker items; or not
>>> resolving them when the code is done, but rather assigning them to
>> the
>>> person responsible for writing the documentation?
>>>
>> Hi,
>>
>> I personally think we should be able to mark bugs which are fixed as
>> being fixed even if we don't yet have tests, otherwise it is a bit
>> misleading (and also, it makes it harder to use QA states like VERIFIED
>> should we want to do that).
>>
>> I think the first question we should ask is if we want to do batched or
>> continuous updates to functional tests and documentation. The batched
>> mode we are currently using does seem to work well as far as I'm aware:
>> a) Documentation-wise, the documentation should provide both a
>> reference and a guide through the program. Guide-type documentation
>> really does need to be batched, because it is highly dependent on how
>> features interact, and we don't want to keep re-writing large chunks of
>> it continuously. Reference type documentation may be easier to write
>> continuously so the developer has fresh memory of all the details,
>> although there is a risk of focusing too much on the technical
>> specifics
>> and not enough on how it would actually be used.
>>
>> b) Functional tests are often like a more comprehensive guide through
>> the program - it is sometimes easier to check that we have all
>> functional tests by enumerating the paths through the UI that you get
>> at
>> them, and this is really what functional tests are supposed to be.
>> Regressions would normally trigger a failure of a functional test
>> created this way if the full functional tests are comprehensive.
>>
>> Of course, automated unit / regression tests are a different story
>> entirely - because they are run so much more often, and test lower-
>> level
>> rather than functional elements, they should be updated for bugs (we
>> really need to get a framework for this set up in PCEnv). In this case,
>> keywords or perhaps flags may be appropriate. We could use flags in the
>> sense that:
>> 1) Someone sees something that they think should have the test-case,
>> so they request the flag.
>> 2) The flag sites there waiting for either a test to be written and
>> the flag +'d, or for consensus that a test wouldn't be appropriate to
>> be
>> reached and the flag minused.
>>
>> Keywords would be another option if we want tracker items to require a
>> test by default - then we could search for those that are resolved
>> without the keyword - we would probably need another keyword to say
>> that
>> it wouldn't make sense to write a test for a given tracker item to make
>> this usable.
>>
>> Best regards,
>> Andrew
>>
>>>
>>> Regards,
>>>
>>> Randall
>>>
>>> ---------------------------------------------------------------------
>> ---
>>> _______________________________________________
>>> cellml-tools-developers mailing list
>>> cellml-tools-developers at cellml.org
>>> http://www.cellml.org/mailman/listinfo/cellml-tools-developers
>>>
>> _______________________________________________
>> cellml-tools-developers mailing list
>> cellml-tools-developers at cellml.org
>> http://www.cellml.org/mailman/listinfo/cellml-tools-developers
>
> _______________________________________________
> cellml-tools-developers mailing list
> cellml-tools-developers at cellml.org
> http://www.cellml.org/mailman/listinfo/cellml-tools-developers



Archive powered by MHonArc 2.6.18.

Top of page