Chronological Thread 

< Chronological >      < Thread >

---

Randall Britten wrote:
> Hi all
>
> I think Andrew has made a very good point regarding why black-box Litmus
> tests are preferable. It is different from the way I usually thought about
> this, but I like it, and accept that we should use it.
>
> Regarding documentation, it seems to me that we still need a way of ensuring
> we don't forget to document new or changed features. I don't have any
> particular preference for how we do this, but I do think we need to have
> some part of our process that ensures this.
>
Hi,

Sorry, I think I probably didn't express my suggestion very clearly
before with respect to documentation (which I think also happens to be
the status quo), so let me clarify how I think this currently works, and
we can use this as a baseline for comparing any suggested changes against:

1) Once in a release cycle, shortly before release, someone (who is
reasonably familiar with the program) reads through the documentation.
2) As they do this, they compare what is documented with what the
program actually does (and have the program open to check this). They
aim to both identify any untrue statements in the current documentation,
and identify any major functionality in the program which is not
mentioned in the manual.
3) The manual is updated based on issues analysed in step 2, and then
this is proof read by several people including the developers for both
grammatical and factual accuracy.
4) Any screen-shots in the manual are updated if they are out of date.

This works well for manuals which are not quite reference documentation
(i.e. don't need to describe every single obscure feature of the
program), but which describe enough of the major features that a normal
user would want to hear about.

I suggest the following factors be taken into account when deciding if
we should change from the status quo process above to another
documentation approach:
=> Overall criterion: Does the benefit with respect to costs increase
or decrease under the new approach?
=> Factors affecting the benefits of documentation:
1) Is the documentation more or less readable for a new user under
the new approach?
2) Is the documentation more or less comprehensive under the new
approach?
3) Is the documentation more or less accurate under the new approach?
=> Factors affecting the costs of documentation:
1) What effect does the new approach have on developer time spent
ensuring documentation get updated?
2) What effect does the new approach have on documentation writer
time spent?
3) How much will the infrastructure for the new approach cost to
implement initially, and to maintain, and how does this compare with the
costs to maintain infrastructure for the current approach?
4) Is the learning curve for new people wanting to contribute to the
project more or less under the new approach?

My answers under these criteria when comparing a 'schedule a
documentation update after each feature update' approach with the status
quo 'review all documentation once per release' would be as follows:
Factors affecting benefits:
1) Is the documentation more or less readable for a new user under
the new approach?
Under a scheduled update, there is a risk that the documentation is less
readable, because it might end up being very fragmented rather than
having a continuous and coherent flow.
2) Is the documentation more or less comprehensive under the new
approach?
The proposed approach will make it possible to have more comprehensive
documentation, because features that the documentation writer might
otherwise miss can be documented.
3) Is the documentation more or less accurate under the new approach?
The proposed approach has aspects which might improve accuracy:
=> Updates to documentation are made more contemporaneously with
updates to the code, and so developer knowledge of the exact semantics
of changes will be more accurate, which might flow on to documentation
accuracy.
However, at the same time, the proposed new approach has aspects which
might harm accuracy:
=> Documentation changes are isolated into lots of small changes
without necessarily having the same level of overarching review, and
such small changes may be too localised in the documentation and not
pick up all consequences of a change.
=> Regularly updating documentation instead of doing it once and doing
it well might also cause people to get sloppy in terms of documentation
quality control.

Factors affecting costs:
1) What effect does the new approach have on developer time spent
ensuring documentation get updated?
If we need to identify documentation changes and flag them on the
tracker item, this will add a little bit of overhead per tracker-item.
2) What effect does the new approach have on documentation writer
time spent?
There will be some benefits in that the documentation writer will be
directed to the changes. However, if documentation is updated on a
fairly continuous basis, the documentation writer will have to regularly
perform these updates, creating more overhead and a greater overall time
commitment.
3) How much will the infrastructure for the new approach cost to
implement initially, and to maintain, and how does this compare with the
costs to maintain infrastructure for the current approach?
The current approach doesn't need much in terms of tracker
infrastructure. The proposed new approach requires a one-off set-up of a
some tracker flags or keywords, which can be done quickly through the
web interface. There is not really any change in the costs to maintain
infrastructure under either approach.

Based on my analysis above, I personally favour the status quo at this
stage. Opinions?

Best regards,
Andrew

> Regards,
> Randall
>
> P.S., I would just like to clarify that I am no fan of bureaucracy and
> burdensome processes, let's keep them as light as possible.
>
>
>> -----Original Message-----
>> From: cellml-tools-developers-bounces at cellml.org [mailto:cellml-tools-
>> developers-bounces at cellml.org] On Behalf Of Andrew Miller
>> Sent: Friday, 9 November 2007 6:00 p.m.
>> To: A list for the developers of CellML tools
>> Subject: Re: [cellml-dev] Documentation and Litmus tests
>>
>> Randall Britten wrote:
>>
>>> My original question leaves open the option of batching docs and
>>>
>> litmus
>>
>>> tests.
>>>
>>> So, we still need to formalise how we ensure that required updates to
>>>
>> docs
>>
>>> and Litmus tests are not forgotten.
>>>
>> I don't think there is a 1-1 mapping between documentation and tracker
>> items, or tracker items and Litmus tests. By batching the tests, we are
>> re-auditing the list of tests / the available documentation every time
>> to ensure that they correspond to the current features visible to the
>> end-user. The way we currently use to do this is by enumerating through
>> every possible distinct UI option that the user has - for example, the
>> user has an option of which top-level menu to choose. We then
>> recursively repeat this process on each sub-menu option, and likewise
>> for every other control available in PCEnv. The person updating the
>> functional tests is expected to know how to test that a control has
>> worked appropriately, but there is usually only one main path to test
>> that.
>>
>> In general, functional tests should be as 'black-box' as possible, and
>> therefore driven from the functional end as I described and not the
>> code
>> end. Many tracker items are likely to come from the functional tests
>> (or
>> something already covered by them).
>>
>> In some cases there can be more complex issues that involve special
>> combinations of performing unusual combinations of actions on different
>> controls. I am not aware of any tracker items which occur in this
>> category, and if there are any, they would certainly be too rare to
>> justify taking some action for every tracker item - instead, perhaps
>> such tracker items could have a keyword set on them. Unless you are
>> aware of a significant number of tracker items needing this, however, I
>> think we should strive to avoid introducing too much bureaucracy into
>> the process - the time costs of following each process adds up quite
>> quickly, and if a certain process has only hypothetical benefits, we
>> are
>> likely to be better off without it.
>>
>> Best regards,
>> Andrew
>>
>>
>>> Regards,
>>> Randall
>>>
>>>
>>>> -----Original Message-----
>>>> From: cellml-tools-developers-bounces at cellml.org [mailto:cellml-
>>>>
>> tools-
>>
>>>> developers-bounces at cellml.org] On Behalf Of Andrew Miller
>>>> Sent: Friday, 9 November 2007 2:44 p.m.
>>>> To: A list for the developers of CellML tools
>>>> Subject: Re: [cellml-dev] Documentation and Litmus tests
>>>>
>>>> Randall Britten wrote:
>>>>
>>>>> Hi all
>>>>>
>>>>>
>>>>>
>>>>> Any preferences as to how we can update our development process so
>>>>> that any time we add a new feature, we have a process for ensuring
>>>>> that the documentation and Litmus tests for these get added?
>>>>>
>>>>>
>>>>>
>>>>> Some possible options: using flags on the tracker items; or not
>>>>> resolving them when the code is done, but rather assigning them to
>>>>>
>>>> the
>>>>
>>>>> person responsible for writing the documentation?
>>>>>
>>>>>
>>>> Hi,
>>>>
>>>> I personally think we should be able to mark bugs which are fixed as
>>>> being fixed even if we don't yet have tests, otherwise it is a bit
>>>> misleading (and also, it makes it harder to use QA states like
>>>>
>> VERIFIED
>>
>>>> should we want to do that).
>>>>
>>>> I think the first question we should ask is if we want to do batched
>>>>
>> or
>>
>>>> continuous updates to functional tests and documentation. The
>>>>
>> batched
>>
>>>> mode we are currently using does seem to work well as far as I'm
>>>>
>> aware:
>>
>>>> a) Documentation-wise, the documentation should provide both a
>>>> reference and a guide through the program. Guide-type documentation
>>>> really does need to be batched, because it is highly dependent on
>>>>
>> how
>>
>>>> features interact, and we don't want to keep re-writing large chunks
>>>>
>> of
>>
>>>> it continuously. Reference type documentation may be easier to write
>>>> continuously so the developer has fresh memory of all the details,
>>>> although there is a risk of focusing too much on the technical
>>>> specifics
>>>> and not enough on how it would actually be used.
>>>>
>>>> b) Functional tests are often like a more comprehensive guide
>>>>
>> through
>>
>>>> the program - it is sometimes easier to check that we have all
>>>> functional tests by enumerating the paths through the UI that you
>>>>
>> get
>>
>>>> at
>>>> them, and this is really what functional tests are supposed to be.
>>>> Regressions would normally trigger a failure of a functional test
>>>> created this way if the full functional tests are comprehensive.
>>>>
>>>> Of course, automated unit / regression tests are a different story
>>>> entirely - because they are run so much more often, and test lower-
>>>> level
>>>> rather than functional elements, they should be updated for bugs (we
>>>> really need to get a framework for this set up in PCEnv). In this
>>>>
>> case,
>>
>>>> keywords or perhaps flags may be appropriate. We could use flags in
>>>>
>> the
>>
>>>> sense that:
>>>> 1) Someone sees something that they think should have the test-
>>>>
>> case,
>>
>>>> so they request the flag.
>>>> 2) The flag sites there waiting for either a test to be written
>>>>
>> and
>>
>>>> the flag +'d, or for consensus that a test wouldn't be appropriate
>>>>
>> to
>>
>>>> be
>>>> reached and the flag minused.
>>>>
>>>> Keywords would be another option if we want tracker items to require
>>>>
>> a
>>
>>>> test by default - then we could search for those that are resolved
>>>> without the keyword - we would probably need another keyword to say
>>>> that
>>>> it wouldn't make sense to write a test for a given tracker item to
>>>>
>> make
>>
>>>> this usable.
>>>>
>>>> Best regards,
>>>> Andrew
>>>>
>>>>
>>>>> Regards,
>>>>>
>>>>> Randall
>>>>>
>>>>> -------------------------------------------------------------------
>>>>>
>> --
>>
>>>> ---
>>>>
>>>>> _______________________________________________
>>>>> cellml-tools-developers mailing list
>>>>> cellml-tools-developers at cellml.org
>>>>> http://www.cellml.org/mailman/listinfo/cellml-tools-developers
>>>>>
>>>>>
>>>> _______________________________________________
>>>> cellml-tools-developers mailing list
>>>> cellml-tools-developers at cellml.org
>>>> http://www.cellml.org/mailman/listinfo/cellml-tools-developers
>>>>
>>> _______________________________________________
>>> cellml-tools-developers mailing list
>>> cellml-tools-developers at cellml.org
>>> http://www.cellml.org/mailman/listinfo/cellml-tools-developers
>>>
>> _______________________________________________
>> cellml-tools-developers mailing list
>> cellml-tools-developers at cellml.org
>> http://www.cellml.org/mailman/listinfo/cellml-tools-developers
>>
>
> _______________________________________________
> cellml-tools-developers mailing list
> cellml-tools-developers at cellml.org
> http://www.cellml.org/mailman/listinfo/cellml-tools-developers
>

---

• [cellml-dev] Documentation and Litmus tests, Randall Britten, 11/09/2007
  • [cellml-dev] Documentation and Litmus tests, Andrew Miller, 11/09/2007
    • [cellml-dev] Documentation and Litmus tests, Randall Britten, 11/09/2007
      • [cellml-dev] Documentation and Litmus tests, Andrew Miller, 11/09/2007
        • [cellml-dev] Documentation and Litmus tests, Randall Britten, 11/12/2007
          • [cellml-dev] Documentation and Litmus tests, Andrew Miller, 11/12/2007
            • [cellml-dev] Documentation and Litmus tests, Randall Britten, 11/12/2007
              • [cellml-dev] Documentation and Litmus tests, Alan Garny, 11/12/2007

---

Archive powered by MHonArc 2.6.18.