During the weekend I did some digging into the quality of measurement, in particular, I tried to answer a question from a colleague on measurement accuracy limits. Well, instead of digging into the accuracy, I managed to look at the validation of measures in general.
I’ve been searching for methods how people evaluate software measures and I came across this nice paper from Laurie Williams and colleagues: https://dl.acm.org/citation.cfm?id=2377661
This systematic review lists 47 criteria used to evaluate software metrics, combining both the empirical and theoretical validation. Here is the list of what they found:
- A priori validity
- Actionability
- Appropriate Continuity
- Appropriate Granularity
- Association
- Attribute validity
- Causal model validity
- Causal relationship validity
- Content validity
- Construct validity
- Constructiveness
- Definition validity
- Discriminative power
- Dimensional consistency
- Economic productivity
- Empirical validity
- External validity
- Factor independence
- Improvement validity
- Instrument validity
- Increasing growth validity
- Interaction sensitivity
- Internal consistency
- Internal validity
- Monotonicity
- Metric Reliability
- Non-collinearity
- Non-exploitability
- Non-uniformity
- Notation validity
- Permutation validity
- Predictability
- Prediction system validity
- Process or Product Relevance
- Protocol validity
- Rank Consistency
- Renaming insensitivity
- Repeatability
- Representation condition
- Scale validity
- Stability
- Theoretical validity
- Trackability
- Transformation invariance
- Underlying theory validity
- Unit validity
- Usability
The list is really impressing, but not all attributes apply to all types of metrics. So, one should always look for the use of metric and then seek the right type of its validation. I recommend this article as great reading for those who are thinking about creating own metrics:)