Software Evaluation Research: Case Study Methodology Designed Research
Project Members/Collaborators: Dr. Seok-Won Lee
Project Description:
A case study design, as an evaluation research approach and a generalization from it, builds a basis for valid inferences from the case study events and evidence collected. For an effective case study as an empirical method applied as a validation exercise – applied to an ‘invented software (systems) engineering methodology’, it is necessary for the validation exercise to first have designed a case study methodology specific to the characteristics of this invented software engineering methodology to be validated using the case study. More specifically, those characteristics are: 1) the characteristics of this invented software engineering methodology that require interventions from the domain Subject Matter Expert (SMEs) or the software engineer in order to perform on demand each appropriate step in the methodology; and 2) the characteristics of the invented software engineering methodology validation that cannot favor the invented methodology over alternatives because of the uniqueness of the invented methodology or the relative different level of understanding of the domain and analytical skill of SMEs in the actual case study ‘experimental’ conditions. Therefore, the consideration of these two above characteristics motivated development of the case study design, based on the theories and guidelines from (Yin, 1994). In addition, the theoretical framework of the case study and the application of this case to the invented methodology will provide better understanding of the methodology, and guidance to researchers from academia and real practitioners from industry.
Case Study Methodology Designed Components
In order to show how and why (“explanatory” type of case study, Yin, 1994) the invented methodology can achieve the research goal, the following five important components (Yin, 1994, p.20) will be defined during the case study design process:
Figure 1 shows the inter-relationships between these five components in explanatory case study design. The study questions can be mapped and further decomposed into a set of study propositions. The developed methodology can then generate results by applying it to units of analysis. The results are linked to the study propositions as evidence through the criteria (using metrics) for interpreting findings.
Figure 1. Case Study Design Components
For example, the goal of the 'PVRD Methodology' research (Lee, 2003) is “the development of a new methodology that can discover missing requirements and reduce the number of incomplete requirements while reorganizing requirements representation space that incorporates and supports multiple viewpoints in legacy status, large-scale, information system requirements specifications (SRSs)”. In order to validate how and why the PVRD methodology can achieve its research goal, the following five important components are designed and defined as follows:
Study Questions: Study questions need to be clarified precisely.
For instance, in this case study design the study question “How and why can
the PVRD methodology discover missing requirements from natural language
software requirements specification documents that involve multiple viewpoints
while reorganizing, improving quality and making a better requirements
representation space?” needs to be clarified by further decomposition of it
into sub questions or propositions. In the dissertation these natural language
software requirements specification documents identified in this study
question are the case study units of analysis.
Study Propositions: Study propositions, derived from the study questions, focus on assertions that should be examined to answer a study question within the scope of the study (i.e. within the scope of the study questions). For instance, in this case study design the general study proposition is “the PVRD methodology works because of its integrated framework (that consists of viewpoints model, enterprise model, missing requirements types categorization model, and requirements discovery and analysis model) and because of the synergism between embedded models and methods that derive mutual benefits”. And four more specific study propositions derived from this proposition are as follows:
- Reduce the number of incomplete requirements by discovering missing requirements.
- Discover requirements defects of various types.
- Discover requirements relationship and workflow process relationship chains in the requirements space.
- Create new requirements indexing structures based on the embedded models.
Units of Analysis: Units of analysis are the selected resources to be examined in the case study. For instance, in the case study methodology designed for this dissertation the PVRD methodology will be applied to the (units) “set of software requirements specification documents” that are expressed in natural language. These requirements (units of analysis) are from the legacy status information-based software system that includes many stakeholders (e.g. interactive systems). Also the Subject Matter Experts (SMEs) and other software development resources (other than SRSs, such as business process descriptions, operational concepts etc.) need to be involved due to their influences in the requirements classification (e.g. assigning requirements to viewpoints) of each model (i.e. Viewpoints Model, Enterprise Model, Missing Requirements Types Categorization Model, Requirements Discovery and Analysis Model) in the PVRD methodology. From the PVRD methodology point of view, each model and method in the PVRD methodology, and the properties associated with the PVRD methodology are the subjects to be examined during a case study using this case study design.
The ‘Requirements Discovery Summary Sheet’(RDSS) is used by a team of SMEs during a case study using this case study design. This RDSS is distributed to SMEs to serve the following roles during a case study:
- Provides a template of how to apply the PVRD methodology to the units of analysis in a case study. Combined with the instructions, SMEs can identify the units that need to be examined (analyzed) and results to be recorded.
- Captures evidence and findings using the criteria defined by metric and measures (combination of qualitative and quantitative analysis) that will be used to support/reject the study propositions. Without having such specific propositions with criteria word attributes in them, an investigator might be tempted to collect “everything” which is impossible to collect (p.22, Yin, 1994).
- Guides SMEs in a step-by-step approach while conducting the PVRD methodology case study. SMEs follow each step of the PVRD methodology and record their findings and observations (unit by unit) under each step in the RDSS.
In addition to collecting specific evidence to support/reject specific propositions, it is also important to collect evidence of the ‘entire process’ of the methodology application to the given requirements set. This is because the specific evidence is captured in the middle or after the application of the methodology, and it did not come from an independent evidence collection process. Also the collection of evidence from the ‘entire process’ of the methodology is used to support/reject the general proposition.
Having the RDSS with clearly identified steps and interpretation criteria (metric and measures) is important and related to the general ‘repeatability’ of the case study methodology.
Linking Data to Propositions: Linking data to propositions represents the data analysis step in the case study design research (Yin, 1994). The PVRD methodology is applied to the units of analysis and plays a role in connecting the generated results to the study propositions.
Figure 2. Linking Data to Study Propositions through Discovery Summary
In a case study using this case study design, the generated results can be based on any mix of qualitative and quantitative evidence. In a case study using this case study design, the generated results will be collected by SMEs in the form of RDSS and the notes from the lessons learned meeting afterwards. In addition, a case study need not always include a direct, detailed observation as a source of evidence (Yin, 1994). In a case study using this case study design, a set of evidence, its analytical interpretations, and lessons learned are sources of evidence. The findings through the RDSS serve as qualitative and quantitative evidence, such as whether or not the discovered missing requirements are defining, mandatory or optional requirements (as qualitative measures defined in the next design component); numbers (quantitative measures) and types of discoveries made, etc.
As an example, Figure 2 shows how the evidence and findings are linked in the support of corresponding study propositions (an explosion of the link between the ‘study propositions’ and ‘linking data to propositions’ in Figure 1). The evidence and findings will be collected through the RDSS by SMEs during a case study.
Criteria for interpreting a case study's findings: Criteria for interpreting a case study’s findings correspond to the metric and measures used in evaluating the results from the properties of requirements defects types defined in the PVRD methodology (such as incomplete, inconsistent, redundant, and ambiguous, as well as requirements relationship chain and workflow process relationship chain). In other words, the results from the requirements discovery and analysis model focus on the findings of significant defects of requirements and improvement of requirements quality as well as the quantitative analysis of how many requirements defects discovered. One example of such findings of significant defects that will be focused on is whether or not the discovered missing requirements or defects are one of defining, mandatory or optional requirements.
The discovery of defining or mandatory requirements is much more critical than discovery of optional requirements. Therefore, the “importance/significance of the discovered requirements defects” serves as a metric in the analysis of the findings and three different requirements types “defining, mandatory or optional requirements” serve as qualitative measures in deciding the significance of the requirements. Also the number of requirements defects of various types serves as quantitative measure. Therefore, the metric and measures will be all interpreted from a combined qualitative and quantitative analysis perspectives based on the summary of RDSS.
All five components in the case study design described guide a case study and become the fundamental basis in validating the developed methodology.
Multiple Case Studies
Some important key points when performing multiple-case studies are summarized from (Yin, 1994). One of the most important points made in Yin’s case study design approach is the design of a theoretical case study framework. Secondly, there is the importance of ‘analytical generalization’ compared to ‘statistical generalization’ in case study design. Survey research relies on statistical generalization, whereas case studies (as with experiments) rely on analytical generalization. In statistical generalization, an inference is made about a population (or universe) on the basis of empirical data collected about a sample (i.e. surveys). In analytical generalization, the investigator is striving to generalize a particular set of results to some broader theory.
The evidence from multiple cases is often considered more compelling, and the overall study is therefore regarded as being more robust (Herriott & Firestone, 1983; Yin, 1994). A theory must be tested through replications of the findings in a second or more case that will lead to an analytical generalization. The logic underlying the use of multiple-case studies is: each case must be selected so that it either 1) predicts similar results (a literal replication) or 2) produces contrasting results but for predictable reasons (a theoretical replication). Under the development of a theoretical framework, a literal replication can explain the conditions under which a particular phenomenon is likely to be found, a theoretical replication can explain the conditions when it is not likely to be found. Each case’s conclusions are then considered to be the information needing replication by other individual cases. For each individual case, the report should indicate how and why a particular proposition was demonstrated (or not demonstrated). Across cases, the report should indicate the extent of the replication logic and why certain cases were predicted to have certain results, whereas other cases if any, were predicted to have contrasting results. Regarding the number of cases in a multiple-case design, since a sampling logic should not be used, the typical criteria regarding sample size also are irrelevant.
Copyright © 2003, Dr. Seok-Won Lee