Table of Contents

Executive Summary

1. Study Background
2. The Fam Vision
3. FAM Concept of Operation
4. Overall FAM Requirements
5. FAM Roadmaps
6. Benefits
7. Conclusions

Appendices

1. Architecture-Based Feasibility Analysis Model

Prepared by: Bob Balzer, Prasanta Bose, and Frank Belz
2. "SIM-PROJECT"

Prepared by: Tom DeMarco
3. Virtual Information SysTem Acquisition (VISTA)

Prepared by: Barry Boehm and Walt Scacchi
4. SAMSA Blue Ribbon Panel Members

Executive Summary

The need for better support technology for software acquisition was recognized by Mr. Lloyd Mosemann, Deputy Assistant Secretary of the Air Force for Computers, Communications, and Support Systems. He also identified Modeling and Simulation (M&S) technology as a strong candidate for such support. M&S technology has been extremely valuable in the acquisition of the hardware parts of Air Force systems, but appears underutilized for software acquisition. Mr. Mosemann commissioned USC-CSE to form a Blue Ribbon Panel to investigate the potential of M&S technology for software acquisition, and to develop a technology roadmap for creating and exploiting any promising technologies identified.

The Panel identified three emerging technology areas which provide the basis for a set of new capabilities worth pursuing:

• Software product architecture-based modeling and simulation;
• Hybrid analytic and dynamic cost and schedule modeling of software projects;
• An acquisition approach featuring the compatible co-evolution of system modeling, development, and instrumentation.

As a result of two workshops, Panel members identified several important FAM requirements including the need to: (i) prototype a software acquisition requirements engineering expert support system (herein called FAM-1) as a demonstration of the FAM concept; (ii) integrate domain-specific product-line architecture and resource and scheduling information into the FAM; and (iii) -term FAM when given a proposed software architecture for program solicitation, to apply FAM to analyze trade-offs and feasibility issues. Panel members also concluded that while useful near-term FAM capabilities were feasible, that full FAM capabilities were a long-range proposition. Thus, an incremental, hierarchical FAM approach is most appropriate.

The hierarchical approach to FAM should initially be focused on addressing the feasibility of top-level or first-cut system requirements, mission objectives, and possible software system architectures in terms of their impact of system reliability, cost, useability, etc. Subsequently, lower-level FAM capabilities should enable more detailed analysis by software system resource and scheduling models/components and later architectural specification components. These elaborations are needed to more fully characterize and analyze the performance, resources, and schedule for a proposed software system's architecture. Later phases of the development of FAM would include incrementally evolving the focus of the FAM requirements from high-level conceptual system components to eventually address proposed or actual architectural product families, components, and implemented/reusable modules for large-scale software systems during their acquisition.

In terms of FAM research and development, the overall Panel consensus was for an incremental approach. In Stage 1, a top-level proof of principle prototype, designated FAM-1, would be developed. Assuming that the prototype convincingly demonstrates the potential value of a FAM capability, Stage 2 would then proceed in two directions. The first direction would produce an initial operational capability, designated FAM-2, incorporating and productizing the most attractive features of FAM-1. The second direction would involve research on high-leverage advanced FAM capabilities, such as architecture-based modeling and multi-attribute tradeoff analysis. Stage 3 would transition maturing research capabilities into downstream FAM-3 increments of capability.

Finally, current conditions increasingly dictate that future systems must be ever more affordable to acquire, while extending the scope of their capabilities and effectiveness.

1. Study Background

The need for better support technology for software acquisition was recognized by Mr. Lloyd Mosemann, Deputy Assistant Secretary of the Air Force for Computers, Communications, and Support Systems, who also identified Modeling and Simulation (M&S) technology as a strong candidate for such support. M&S technology has been extremely valuable in the acquisition of the hardware parts oft Air Force systems, but appears underutilized for software acquisition. Mr. Mosemann commissioned USC-CSE to form a Blue Ribbon Panel to investigate the potential of M&S technology for software acquisition, and to develop a technology roadmap for creating and exploiting any promising technologies identified.

The Panel identified three emerging technology areas which provide the basis for a set of new capabilities worth pursuing:

• Architecture-based modeling and simulation for software systems (see Appendix 1);
• Hybrid analytic and dynamic cost and schedule modeling of software projects (see Appendix 2);
• An acquisition approach featuring the compatible co-evolution of system modeling, development, and instrumentation (see Appendix 3). Each of these three technology areas are elaborated in Appendices 1, 2, and 3 respectively, and the Panel participants are listed in Appendix 4.

Through two workshops held respectively in August 1995 and October 1995, the Panel members examined, debated, and then identified a consensus for how modeling and simulation tools and techniques might best be applied to support software acquisition. Specifically, the Panel recommended a research and development agenda addressing both near-term and longer-term strategies. Key to launching and cordinating the overall effort was the recomendation to focus initial attention at the rapid prototyping and multi-stage iterative development of a feasibility analysis modeling system herein called FAM, or FAM-1 in its proposed first stage.

The remainder of this report thus outlines the FAM vision and requirements, the technical approach to its development, and a set of technology road maps that identify the role and sequence of technological opportunities to pursue in order to most efficiently achieve the results.

2. The FAM Vision

In order to address this uncertainty, the user, acquirer, or developer organizations would benefit from the ability to follow a stepwise approach that allows them to analyze the feasibility of alternative functionality configurations. Similarly, such an approach should employ a hierarchical tool set that supports the feasibility analysis by providing more robust assessments as the system's definition continues to be refined and evolved. In this way, the stepwise approach and hierarchical tool set would support flexibility in focusing on key technical or resource issues that may appear at different levels of system requirements and architecture. Similarly, after system development starts, proposed changes, adjustments, or additions to system requirements ("creeping requirements") could be subjected to a sensitivity analysis to determine their impact on feasibility components such as cost, schedule, staffing, functional architecture, and other risk areas before commitment is made for whether or not to incorporate them into the ongoing development and acquisition effort.

The basic premise that underlies the technical agenda for FAM is that determining the compatibility of software system plans, requirements, and architecture is the key to determining feasibility for acquisition and development. We believe this is true whether addressed from the perspective of system users, acquirers, or developers. However, we note for each of these separate perspectives, that different issues and concerns regarding software requirements and architecture may need to be addressed. Nonetheless, it can be constructive to envision the needs of these different audiences being addressed through a common, shared, and open set of FAM tools and techniques, followed by more capable set of modeling and simulation tools that help to articulate, understand, and analyze evolving software system requirements and proposed implementation architectures.

The basic problems to solve through FAM technology are:

• How to determine the risks and feasibility of a proposed software development? This is a near-mid term (2-3 year) objective. It necessitates the acquisition and use of benchmark test cases or knowledge-based domain models to calibrate the FAM. When needed data is not available, then techniques and computational methods for sensitivity analyses will need to be employed.
• Beyond this, we want the capability to elaborate the feasibility model in order to support technology development or implementation. This is a mid-long term (5-7 year) objective. Such elaboration requires further research as well as periodic demonstrations of emerging research capabilities.

Overall, the Panel concludes that the acquisition process can be improved by the use of the FAM, either short-term or long-term. The focus of this report is primarily on FAM technologies. In addition, however, a more ambitious long-term process approach for applying modeling and simulation technology to software acquisition is provided in the Extended Report's Appendix 3 on Virtual Information System Acquisition (VISTA).

3. FAM Concept of Operation

• Concept Feasibility Determination: Given a new mission or strategic objectives, determine whether appropriate technology, architectures, and resources can be feasibly brought together into a target software system application in an affordable and timely manner.
• Architecture Feasibility Determination: Given a proposed software application architecture, determine whether it can satisfy mission or strategic objectives in an affordable and timely manner.
• Virtual System Acquisition: Given a feasible application concept and architecure, acquire the proposed architecture as a series of modeled, simulated, or implemented subsystems that can be incrementally developed in the course of progressively replacing or transforming the modeled or simulated subsystems with real implementations.
• FAM-1: Top-Level Feasibility Advisor, Parametric Models: In order to determine whether the candidate technologies, architectures, and resources can be brought together in a feasible manner to address a new mission or strategic objectives, a top-level feasibility analysis modeling system, FAM-1, can be used to check established acquisition heuristics and parameters.
• FAM-2: SW-Intensive Models and Simulations: In order to determine whether a proposed application system architecture is viable, an environment for software-intensive modeling and simulation, FAM-2, can be used to prototype, analyze, and execute system architectural capabilities and functionality, then reconcile these performance characteristics against cost, schedule, and quality trade-offs among proposed architectural design alternatives.
• FAM-3: Hybrid Measurement, Modeling and Simulation Environment: In order to acquire an incrementally developed software application system, a hybrid FAM-3 environment can be used which cooperatively models, simulates, and measures the performance capabilities of an evolving application system, its subsystems, and their collective architectural design.
• SW Feasibility Heuristics: In order to provide plausible advice for how to assess the top-level feasibility of an emerging software application system, we collect, validate, and refine a knowledge base of heuristics that help determine what matters and what technology, architecture, or resource characteristics affect the overall feasibility of the system.
• Arch. representation and analysis M&S. Advanced cost/schedule/quality M&S: In order to rapidly and affordably construct software application systems, we need to research and develop new tools and techniques for incrementally building and evolving large application systems using models or simulations of the application's subsystems, together with the incorporation of already implemented or newly development components.
• Integration into commercial SEE extensions: In order for FAM tools to be broadly applied across the spectrum of Air Force acquisitions, they need to become available as extensions to currently available software engineering environments.

• Pre-proposal Requirements Analysis: Analyze feasibility of program requirements (a sanity check on the technical perspective for a new program to determine rough order of magnitude for cost, architecture, other risk items, etc.) prior to RFP.
• Proposal Analysis: Upon receipt of contractors' proposals, analyze each proposal for feasibility, determine which proposals are in competitive range, and what assistance is needed to evaluate the technical perspective (e.g., architecture) of those proposals within competitive range.
• Project Start-up: Evaluate the feasibility of resources (cost, people, etc.) and schedule of proposed system design. This could also be used for "fly-off" scenarios as well, when competing designs are being evaluated.
• Ongoing Program Review: Re-analyze feasibility at milestones during development life cycle, as well as when significant program or system requirements changes occur.

FAM should be applicable to product-line software system architectures, as well as to unique non-product-line software systems. It appears that the FAM may be more readily suited to product-line software system architectures, since their recurring development can accomodate both the collection, refinement, and recalibration of the FAM for the product-line's application domain. However, it may also be useful for (portions of) non-product-line software, especially where the software is defined by a well-conceived reference model standard, such as the emerging Air Force Horizon Architecture. Nonetheless, within the domains of C3I and other Air Force applications, we may expect future systems to be more likely to follow a product-line architecture, since industry trends and corporate strategies may lead system development contractors to focus their expertise and core competencies around the mastery of product-lines, rather than individual products or contracts.

4. Overall FAM Requirements

• Through a series of open Panel discussions and working group reports, a number of explicit and implicit requirements for the FAM were identified. These requirements address how the FAM should accomodate data and analyses pertaining to software requirements and their relation to software systems architecture, software development resources, schedules, and processes, as well as related technological opportunities. In most general terms, FAM initially should operate as a requirements "short-list" engineering expert support system. More specifically, this implies that FAM should support:
• The asssessment of operational software system requirements that are quantifiable and testable, such that checklists can be presented in a graphic user interface to assess software system requirements in their formative stages.
• When characteristics of software system requirements can be assigned quantified values,the FAM should record on what basis the values are motivated by the program mission or military threat, as another way to "test" requirements appropriateness.
• The FAM should assess domain-independent characteristics of software system requirements (e.g., large number of interfaces to legacy/COTS systems; uncertain system performance requirements that exclude classes of possible system solutions or architecture; accuracy or precision requirements that do not add to military effectiveness; too much power or weight required to support the infrastructure (platform or information technology) needed to effectively operate the software system; very high display bandwidth that exceeds human perceptual capabilities; reliance on a single performance algorithm; etc.).
• The FAM should facilitate weighting and prioritizing of requirements.
• The development of the FAM should employ a facilitator or contractor to elicit/acquire characteristics of large-scale software system requirements that are most likely to lead to successful or unsuccessful development efforts.
• The FAM should provide interactive graphic output presentations that display potential trade-offs, consequences, cost alternatives and the like that highlight feasibility, rough order of magnitude cost, etc.
• The FAM should provide capabilities that enable a form of software acquisition and development project simulation, patterned on alternative planning scenario and wargame-like exercises, such as those addressable in COTS products like "SimCity".
• The FAM should support continuous or ongoing use in support of software acquisition programs: from early pre-proposal activities through program completion.
• The FAM should provide a graphic user interface that provides for direct manipulation of input constraints and output values.
• The FAM should provide risk assessment and management capabilities.
• The FAM should support end-users in requirer, acquirer, and developer organizations.
• For FAM to become effective in improving software acquisition practices, we must also consider how to motivate the use of the FAM by senior management. Apparent strategies to pursue to address this include:
• The FAM should be transparent ("white box") technology that is relatively easy to explain to senior management in User, Acquirer, and Developer organizations. Eventually, FAM should be able to explain its decisions down to the data level.
• The development of FAM should provide a mechanism or interactive documentation that provides an index for the kinds of questions that can be answered through FAM.
• There should be an independent FAM organization/expert, who can act as a consultant or advisor to SPO's, among others.
• Identifying incentives for the adoption and use of FAM in User, Acquirer, and Developer organizations, as well as identifying the constraints or barriers to overcome in facilitate adoption and use of FAM as a "best business practice."

Development of the FAM should proceed incrementally, incorporating more sensitivity to (a) requirements analysis and (b) architecture as the capabilities emerge from concurrent research activities. In this regard, the VISTA approach in Appendix 3 should be applied to development of FAM, so that the FAM is developed through a series of ever more complete and operational system modeling and simulation software components.

Development of the FAM should also undertake a number of activities concurrently, including:

• Overall coordination of the FAM project.
• Development of advanced cost estimation models reflecting new acquisition processes, COTS and product line architectures, and rapid application composition capabilities.
• Evaluation of existing requirements analysis tools. We need to compare the effectiveness of available requirements analysis tools and techniques for use in FAM.
• Research towards the development of an executable architecture analyzer. Our vision is that it is desirable to accomodate executable software architectures in FAM, whereas in the Elaboration follow-on, it is considered mandatory.

5. FAM Roadmaps

The consensus of the Panel was that the hierarchy of increasingly detailed FAM's would be applied concurrently with the prototyping and incremental development of a new Air Force system during its acquisition. During this system elaboration, the critical properties of the prototypes and increments would be measured and used both for system feasibility validation and for calibration of an increasingly accurate FAM for the system being developed. After initial system deployment, the system measurement and FAM calibration process would continue through the system's evolution.

In terms of FAM research and development, the overall Panel consensus was for an incremental approach. The results of the initial stage, Stage 0, are represented by this report as a result of effort during 1995. In Stage 1, during 1996, a top-level proof of principle prototype, designated FAM-1, would be developed. Assuming that the prototype convincingly demonstrates the potential value of a FAM capability, Stage 2 would then proceed in two directions. The first direction would produce an initial operational capability, designated FAM-2, incorporating and productizing the most attractive features of FAM-1. The second direction would involve research on high-leverage advanced FAM capabilities, such as architecture-based modeling and multi-attribute tradeoff analysis. Stage 2 thus spans 1997-1998. Stage 3 would then incrementally transition maturing research capabilities into the downstream FAM-3 system in the years 1999-2001. Figure 2 organizes and outlines these stages, together with the proposed activities to be performed.

FAM-related Software Architecture Technology Road Map

• Develop abstraction hierarchy of FAMs where the software product model at any level is a partial model of the actual code and feasability prediction at each level is more detailed than the level above.
• Develop parameterized architecture-based models where architecture-level elements (at each level of FAM abstraction) gets modelled using a set of parameters that define the principal drivers of different quality attributes
• Develop collaborative and risk-driven process models for distributed engineering of heteregeneous software process that utilizes the FAM abstraction hierarchy representation of software to coordinate decisions and negotiate tradeoffs.
• Develop architectural domain models and associated knowledge bases for C3I and other Air Force applications (Stage 2 and Stage 3).
• Investigate and develop analytical principles for making and evaluating trade-offs for alternative/integrated models of software system architectures, together with the resources and schedules allocated for their implementation (Stage 2 and Stage 3).
• Develop architecture templates or domain-specific kits that allow software architectures for C3I and other Air Force applications to be specified by selecting values for relevant architectural paramaters or variables (Stage 2 and Stage 3).
• Develop various "-ility" models (for reliability, dependability, maintainability, reusability, deliverability, usability, testability, etc.) that assist in determining the credibility of a proposed software architecture for a C3I or similar Air Force application (Stage 1 through Stage 3).
• Develop tools and techniques for specifying, analyzing, interpreting, or otherwise reasoning about partial/complete executable software architectures appropriate for C3I and other Air Force applications (Stage 2 and Stage 3).

FAM-related Software Resource and Scheduling Technology Road Map

• Stimulate the development and Air Force calibration of updated versions of analytic cost and schedule models such as COCOMO, Softcost, PRICE S, Checkpoint, SEER, etc. (Stage 0 through Stage 2).
• Integrate COTS tools providing PERT (program evaluation and review technique) and CPM (critical path method) techniques with these analytic cost and schedule modeling tools (Stage 1 and Stage 2).
• Develop or incorporate business case analysis tools, such as Turbo FEAM (functional economic analysis model) and Turbo BPR (business process reengineering) developed and deployed by DISA (Stage 1 and Stage 2).
• Develop user-oriented software requirements elicitation and analysis tools and techniques (Stage 1 through Stage 3).
• Develop and integrate multi-model resource and scheduling engines (e.g., for System Dynamics models, discrete-event models, knowledge-based models) for analysis and simulation computations (Stage 2 and Stage 3).
• Develop or integrate heuristic re-resourcing and re-scheduling mechanisms that can perform their computations incrementally (Stage 1 and Stage 2).

FAM-related Technology Road Map

• Survey and assess existing application specific capabilities and tools that successfully address specific analysis tasks like performance analysis, reliability analysis, and etc. Develop generic models - obtained via generalization of the domain specific analysis tools for single quality attributes. Develop models of relations between quality attributes.
• Survey and assess the capabilities of existing and near-term software system requirement analysis and risk assessment tools, including those in the research community, and those available as COTS products (Stage 1 and Stage 2).
• Develop a requirements "short-list" engineering expert support system that can serve as a user environment, similar perhaps to the popular "SimCity" game available for PCs (e.g."SimProject Assistant" (SPA)) (Stage 1).
• Support operational requirements of new system acquisitions that are quantifiable and testable, and to which identifiable sanity checklists can be applied. When requirements can be assigned quantified values, determine or assess on what basis the values are well motivated by the program mission or military threat, as another way to "test" requirements validity (Stage 2 and Stage 3).
• Provide mechanisms or user interfaces to help detect problematic, domain-independent requirements (e.g., large number of interfaces to legacy/COTS systems; uncertain system performance requirements that exclude classes of possible system solutions or architecture; accuracy or precision requirements that do not add to military effectiveness; too much power of weight required to support the infrastructure (platform or information technology) needed to effectively operate the software system; very high display bandwidth that exceeds perceptual capabilities; reliance on single performance algorithm;...) (Stage 1 and Stage 2).
• Provide support for weighting and prioritizing of requirements (Stage 1 and Stage 2).
• Employ SPA facilitator (e.g., contractor) to elicit/acquire domain-independent requirements "short-list" (or Red-Team) expertise (Stage 1).
• Develop graphic user interfaces that display potential trade-offs, consequences, cost alternatives and the like that highlight feasibility, rough order of magnitude cost, etc. (Stage 1 through Stage 3).
• Manage uncertainties associated with emerging system requirements for possible or proposed software system architectures; for example, keeping track of what parts of the emerging system have been architected (versus not yet designed), and of the designed part, which part has been modeled (and to what "level of detail"), and what has been coded (or reused) (Stage 1 through Stage 3).
• Integrate domain-specific product-line architecture and resource and scheduling information into SPA (Stage 2 and Stage 3).
• Incorporate application domain-specific concerns (e.g., function points expected for a C3I application) (Stage 2 and Stage 3)
• Add "game designer" to SPA development to orchestrate SPA's concept of operation (Stage 2 and Stage 3).
• Provide "cockpit" based user interface to SPA game environment (Stage 2 and Stage 3).
• Given a proposed software architecture for program solicitation, apply SPA to re-analyze software requirements to re-evaluate possible trade-offs, feasibility displays, etc. (Stage 2 and Stage 3).
• Develop tools supporting the navigation and selection of risk analyses indicated by the SEI Risk Taxonomy (Stage 1 and Stage 2).
• Develop form-based hypertext graphic user interfaces and navigational browsing mechanisms for soliciting and displaying graphic results (e.g, over the Internet or World Wide Web) from FAM-related computations (Stage 1 and Stage 2).
• Develop or incorporate software risk assessment reasoning engines and rule bases that support the querying and explanation of FAM analyses, trade-offs, and overall capabilities (Stage 1 and Stage 2).
• Develop and integrate process guidance, replay, and feedback mechanisms that help instruct unfamiliar users in the use of FAM tools and techniques, as well as to help enable the continuous improvement of the FAM process (Stage 1 through Stage 3).

6. Benefits

• analyze feasibility, risk, and affordability at various points in the software acquisition life cycle
• reduce software acquisition risks and overruns
• explore more system options and develop more capable and flexible systems
• respond more rapidly and effectively via software to information warfare threat situations
• train future AF personnel to be smart software buyers and users, and
• utilize advances in FAM-related modeling and simulation technologies, software architecture principles and technologies, as well as software resource and scheduling technology to more effectively master the use of cyberspace in support of new system acquisition.

Figure 3 shows a tangible example of the kind of benefit that could be provided by an architecture-based FAM capability. In the 1980's, a large government system went into acquisition with a requirement for a one-second response time. Without a strong FAM capability, the best the project could do was to develop the highly complex Architecture A. This architecture involved over 20 super midi-computers caching data for rapid access by the system's 2000 concurrent users. It would barely meet the one-second response time requirement, at a cost of over $100M.

With a FAM capability, alternative architectures could have been analyzed. The agency could have been presented with the Architecture B alternative: a $30M system with a four-second response time. By employing user prototyping (a step the project eventually took, about two years into the acquisition), the agency could have determined right away that Architecture B provided an acceptable capability at far less cost and risk.

---

Figure 3. Example of Need for Architecture-based Modeling

---

7. Conclusions

The Panel concludes that the time and technology for applying modeling and simulation in support of software system acquisition is at hand, and that the Air Force is positioned to take advantage of the opportunities and benefits that can result. The evolutionary development of a feasibility analysis modeling system now follows as the next logical step, using the staged Roadmap provided in this report. The resulting capabilities will enable new mission-critical Air Force systems to continuously calibrate and maintain their feasibility throughout a progressive virtual-into-real acquisition.

Appendices