Knowledge-Based Software Process (Re)Engineering:
An experience report

This presentation can be found on the WWW at the URL:
http://www.usc.edu/dept/ATRIUM/Software_Process.html

Walt Scacchi, ATRIUM Laboratory, Information and Operations Management Dept., School of Business Administration, and Center for Software Engineering, University of Southern California. 213-740-4782, 213-740-8494 (fax) (Scacchi@gilligan.usc.edu).

Copyright © 1997 Walt Scacchi, The University of Southern California. All RIGHTS RESERVED.

Overview

Background and Definitions

Meta-Modeling

Definition and Modeling

Analysis

Simulation

Visualization

Prototyping, Walkthrough, and Training

Administration: Staffing and Scheduling

Integration: Data and Systems

Target Support Environment Generation

Instantiation and Enactment

Monitoring and Measurement

Enactment History Capture and Replay

Articulation

Evolution

In-Progress Efforts

Conclusions

A paper providing an overall description of this process engineering life cycle can be found at

http://cwis.usc.edu/dept/ATRIUM/Papers/Business_Process_Modeling.ps.

Software development into the 21st. Century

Accomodates distributed everything
Assumes legacy systems, architectures, and frameworks (needing reengineering?)
Employs "commercial-of-the-shelf" components
Incorporates software agents, applets, mediaware, and plug-ins
Utilizes "glue-code" or middleware
Accomodates dynamic (re)configuration of systems or components
Supports old and new data sets, formats, models, and repositories
Incorporates models and simulations of subsystems
Operates as a "virtual enterprise" utilizing an electronic communication infrastructure
Behaves as an open system with dynamically breaking components.

Return to Overview

Background and Definitions

Our focus is targeted at the engineering of complex business processes or capabilities across their life cycle.
A capability represents the processes, organization staffing, and information infrastructure, as well as their interrelationships, for a recurring business activity that produces products or services.
The web of relationships among the objects and attributes of a product, process, organization, infrastructure, or total capability defines its architecture.

A paper providing an overall description of a software development capability architecture can be found at

http://cwis.usc.edu/dept/ATRIUM/Papers/Process_Meta_Model.ps

The overall architectural strategy for a software production infrastructure that supports the process life cycle was previously presented in the paper W. Scacchi, "The Software Infrastructure for a Distributed System Factory," IEE Software Engineering Journal,, Vol. 6(5), 355-369, (1991).

Another paper which surveys architectural features of more than 60 software environments can be found at

http://cwis.usc.edu/dept/ATRIUM/Papers/MetaCASE.ps

Return to Overview

Sources of Experiences Encountered

Andersen Consulting
AT&T Bell Laboratories
CoGenTex
CTA
EDS
Hewlett-Packard
HoloSoFx (formerly Active Management Inc.)
McKesson Water Products Co.
MCC
Naval Air Warfare Center (China Lake, CA)
Northrop(-Grumann) B-2 Division
Office of Naval Research
Perceptronics
SUN Microsystems Computer Corp.
USAF Rome Laboratories
Return to Overview

Meta-Modeling

Constructing and refining a process concept vocabulary and logic (an ontology) for representing families of capabilities, processes, and their instances in terms of object classes, attributes, relations, constraints, control flow, rules, and computational methods.
Designed to accomodate distributed multi-agent (w/multi-task agendas) problem-solving in the "domain" of process.
Experience: key to achieving process-level interoperability, as shown with ERA Product-centered DB (PBI-Softman), Attributed Petri Nets (CACE-PM/DMS*), Rule-Based Databases (AP5, Matisse), Process Programming Language (SynerVision*), Workflow (WORKFLOW/BPR*), Hybrid Composite (SMART), Others {PIF}.
```
* denotes commercial product.
```
Image files that show user interface displays of (a) a meta-model class hierarchy we use, and (b) a meta-model schema for the "task-force" class can be viewed when selected.

A paper providing a more detailed description of meta-modeling can be found at
http://cwis.usc.edu/dept/ATRIUM/Papers/Process_Meta_Model.ps.
Return to Overview

Definition and Modeling

Eliciting and capturing of informal process descriptions, and their conversion into formal process models or process model instances.
Experience: Most "as-is" processes are ill-defined and not well understood.
Experience: Most process redesign efforts want to primarily focus on "to-be" alternatives, without baselining as-is processes.
Experience: Transitioning from as-is to to-be processes is itself a process that must be understand, modeled, and operationalized.
Image files that show user interface displays of (a) a process task model class hierarchy definition conforming to MIL-STD-2167A, and (b) a process model definition for a programming task in a table format can be viewed when selected.
A description of the definition format and mechanism can be found in the paper P.K. Garg, P. Mi, T. Pham, W. Scacchi, and G. Thunquest, "The SMART Approach to Software Process Engineering," Proc. 16th. Intern. Conf. Software Engineering,, IEEE Computer Society, Sorrento, Italy, 341-350, (1994).

Return to Overview

Analysis

Logical:Evaluating static and dynamic properties of a process/capability model, including its consistency, completeness, internal correctness, traceability, as well as other semantic checks.
Reasoning: Query-based pattern-matching and inference to reason about space, time, organization (i.e., who, what, where, when, why, how, what-if, plus relational operators and filters), classification (taxonomic, genericity), configuration (composition, scheduling, replanning, generalization, specialization), and diagnosis (articulation).
Feasibility: Determining whether a proposed process or capability architecture can satisfy existing requirements, given available resources.
Optimization:Determining how to transform process flow to reduce resource utilization (e.g., reduce cycle time and cost).
Experience: Best source of high-value, short-term results and payoffs.
Experience: Easy to produce management reports or presentation materials from analysis mechanisms.
Image files that show user interface displays of (a) a sample of process model analysis checks, (b) process model analysis statistics, and (c) process model analysis view, can be viewed when selected.

Return to Overview

Simulation:

Knowledge-Based Simulation

Symbolically enacting process models in order to determine the path and flow of intermediate state transitions in ways that can be made persistent, replayed, queried, dynamically analyzed, and reconfigured into multiple alternative scenarios.
Experience: High-value technology is infrequently used.
Experience: Can produce narrative summaries of simulation runs.
Image files that show user interface displays of (a) a process model simulation interaction, and a subsequent (b) a process model simulation narrative trace can be viewed when selected.
A paper describing the initial design and implementation of this simulation mechanism can be found in, P. Mi and W. Scacchi, "A Knowledge-Based Environment for Modeling and Simulating Software Engineering Processes," IEEE Trans. Knowledge and Data Engineering, Vol. 2(3), 283-294, (1990). Reprinted in Nikkei Artificial Intelligence, Vol. 20(1), 176-191, (1991, in Japanese).

Discrete-Event Simulation

Computationally enacting a sample of process models as network flows with hueristic or statistical arrival rates and service times so as to determine the overall process performance envelope, throughput, systematic behavior, and resource bottlenecks.
Experience: Although less flexible, easy to use to discover process optimizations.
Experience: Visual interactions and presentations always impress.
Image files that show user interface displays of (a) a discrete-event process model workflow simulation interaction, and a subsequent (b) simulation results display highlighting distribution of costs and activity-based costs can be viewed when selected.

Return to Overview

Visualization

Providing users with graphic views of process models and instances that can be viewed, navigationally traversed, interactively edited, and animated to convey process statics and dynamics.
Experience: Process visualizations enable intuitive analysis and discovery.
Experience: Visualization appears key to acceptability.
Image files that show user interface displays of (a) visual process model editor, (b) spreadsheet-like process browser, (c) graphical process model browser, and (d) graphical process object browser can be viewed when selected.

Return to Overview

Prototyping, Walkthrough, and Performance Support (Training On Demand)

Incrementally enacting partially specified process model instances in order to evaluate process presentation scenarios to end users, prior to performing tool and data integration.
Experience: Process prototyping and walkthrough is effective enabler for eliciting user feedback.
Experience: Can provide a basis for user empowerment in controlling design and improvement of local processes.
Experience: Generation of performance support materials in response to process improvements or changes is well-received.
Image files that show user interface displays of (a) a process prototype or walkthrough display, and (b) a view of automatically generated process performance support documentation produced in HTML can be seen when selected.

Return to Overview

Administration: Staffing and Scheduling

Assigning and scheduling specified users, tools, and development data objects to modeled user roles, product milestones, and development schedule.
Experience: Incremental and hueristic rescheduling functions always impress managers.
Experience: This demonstrates scheduling flexibility that may not be available in other tools.
Image files that show user interface displays of (a) a process action precedence order to be scheduled, (b) an initial staff schedule assignment, and (c) an optimized schedule assignment after process analysis and improvement transformations, which can be viewed when selected.

Return to Overview

Integration: Data, Tool, User Interface

Provides a client-side workspace as a computational context (a cached web binding user, org-role, task, tool(s), input and output resources (with "status")) at the granularity of enactable actions (i.e., process atoms -- semantic units of work).
Entails encapsulating or wrapping selected information systems, repositories, and data objects that are to be invoked or manipulated when enacting a process instance.
Experience: Can entail a lot of difficult technical work, but its relatively easy to finesse when constructing "concept demo's."
Experience: Growing interest in providing support for integration of wide-area heterogeneous information repositories using the Internet.
Image files that show user interface displays of (a) multiple tools bound to process actions that are integrated with underlying object manager (not shown) via software broadcast message server (not shown).
A paper describing the strategy and mechanisms supporting process integration can be found in the paper, P. Mi and W. Scacchi, "Process Integration for CASE Environments," IEEE Software, Vol. 9(2), 45-53 (1992). Reprinted in Computer-Aided Software Engineering, 2nd. Edition, E. Chikofsky (ed.), IEEE Computer Society, (1993).

A paper describing the mechanisms support data repository integration can be found at
http://cwis.usc.edu/dept/ATRIUM/Papers/Integrating_Software_Repositories.ps.

A paper describing the mechanisms support data repository integration with adaptive process enactment within a virtual software development enterprise can be found at
http://cwis.usc.edu/dept/ATRIUM/Papers/DHT-95.ps.

Return to Overview

Target Support Environment Generation

Automatically transforming a process model or instance into a process-based computing environment that selectively presents prototyped or integrated information systems to end-users for process enactment.
Experience: Considered a unique capability, not available in other process environments.
Experience: Simplifies or eliminates low-level process programming via "application generator" techniques.
Image files that show user interface displays of (a) process-encapsulated tool environment that was generated via automated transformation of the modeled and integrated process.
A description of this mechanism can be found in the paper P.K. Garg, P. Mi, T. Pham, W. Scacchi, and G. Thunquest, "The SMART Approach to Software Process Engineering," Proc. 16th. Intern. Conf. Software Engineering,, IEEE Computer Society, Sorrento, Italy, 341-350, (1994).

Return to Overview

Instantiation and Enactment

Performing the modeled process using the environment by a process engine that guides or enforces specified users or user roles to enact the process as planned.
An example display of a modeled process during execution.
Process tasks or actions have "status", which get propagated within context, and across contexts when a context is closed (check-in). (NB: DHT provides a different strategy, which binds status values to resources, rather than to only process tasks).
We provide a "process enforcement policy variable" that allows progressive relaxation of process enactment constraints (e.g., relax process step or product pre-conditions). This supports process maturation, but also increases likelihood of process breakdowns.
Where automation of modeled process action occurs, the user does not see it, except for process task/action status propagation display.
Return to Overview

Monitoring and Measurement

Collecting and measuring process enactment data needed to improve subsequent process enactment iterations, as well as documenting what process actions actually occurred in what order.
Experience: Another feature found very attractive by managers.
Experience: Key source of data for process improvement, optimization, or evolution.
A paper describing different types of process measurements of interest can be found at
http://cwis.usc.edu/dept/ATRIUM/Papers/Process_Measurement.ps.
Return to Overview

Enactment History Capture and Replay

Graphically simulating the re-enactment of a process, in order to more readily observe process state transitions or to intuitively detect possible process enactment anomalies.
Experience: Visualizing and replaying process enactment histories is well-received by managers and executives.
Experience: Supports "organizational drill-down" when process anomalies are observed.
An image file that shows a user interface display of (a) process enactment event history and timing measurements can be viewed when selected.

Return to Overview

Articulation

Diagnosing, repairing, and rescheduling actual or simulated process enactments that have unexpectedly broken down due to some unmet process resource requirement, contention, availability, or other resource failure (currently diagnozes 70 classes of failure, and navigates 300 repair hueristics).
Experience: A research result that is ahead of its time.
Image files that show user interface displays of (a) a class hierarchy of articulation diagnosis and repair mechanisms, which can be viewed when selected.
A paper describing the design and implementation of the process articulation support system can be found in the paper P. Mi and W. Scacchi, " Modeling Articulation Work in Software Engineering Processes," Proc. 1st. Intern. Conf. Software Process, IEEE Computer Society, Redondo Beach, CA (1991).

Return to Overview

Evolution: Continuous Improvement and Model Management

Incrementally and iteratively enhancing, restructuring, tuning, migrating, or reengineering process models and process life cycle activities to more effectively meet emerging user requirements, and to capitalize on opportunitistic benefits associated with new tools and techniques.
Experience: One of the few process model repositories, which also accomodates process formalization and knowledge-based operations.
Experience: Another research result ahead of its time.
Image files that show user interface displays of (a) a process class hierarchy for organizing related process models, which can be viewed when selected.

A paper describing the design and implementation of a knowledge-based process repository supporting these capabilites can be found in P. Mi, M.J. Lee, and W. Scacchi, "A Knowledge-based Software Process Library for Process-Driven Software Development," Proc. 7th. Annual Knowledge-Based Software Engineering Conference, IEEE Computer Society, Washington, DC, pp. 122-131, (September 1992)

Return to Overview

Conclusions

Software process engineering is a dynamic team-based endeavor that can only lead to mature processes through rapid process prototyping, incremental development, iterative refinement, and the reengineering of ad hoc process task instances and models.
Process/capability engineering may be most likely to succeed when focused on high frequency or short cycle-time processes.
New techniques for rapid process design, trade-off analysis, and customization are needed.
There are "pathological" software development processes that are resistant to systematic (re)engineering, and thus should be avoided.

In-Progress Efforts

Return to Top

This interactive presentation page is maintained by Walt Scacchi who can be reached at the e-mail address noted above. This page was last updated on 22 April 1997.

Copyright © 1997 Walt Scacchi, The University of Southern California. All RIGHTS RESERVED.

A paper providing an overall description of this process engineering life cycle can be found at

Software development into the 21st. Century

A paper providing an overall description of a software development capability architecture can be found at

Sources of Experiences Encountered

Image files that show user interface displays of (a) a meta-model class hierarchy we use, and (b) a meta-model schema for the "task-force" class can be viewed when selected. A paper providing a more detailed description of meta-modeling can be found at

Image files that show user interface displays of (a) a sample of process model analysis checks, (b) process model analysis statistics, and (c) process model analysis view, can be viewed when selected.

Image files that show user interface displays of (a) a discrete-event process model workflow simulation interaction, and a subsequent (b) simulation results display highlighting distribution of costs and activity-based costs can be viewed when selected.

Image files that show user interface displays of (a) visual process model editor, (b) spreadsheet-like process browser, (c) graphical process model browser, and (d) graphical process object browser can be viewed when selected.

Image files that show user interface displays of (a) a process prototype or walkthrough display, and (b) a view of automatically generated process performance support documentation produced in HTML can be seen when selected.

Image files that show user interface displays of (a) a process action precedence order to be scheduled, (b) an initial staff schedule assignment, and (c) an optimized schedule assignment after process analysis and improvement transformations, which can be viewed when selected.

A paper describing the mechanisms support data repository integration with adaptive process enactment within a virtual software development enterprise can be found at

A paper describing different types of process measurements of interest can be found at

An image file that shows a user interface display of (a) process enactment event history and timing measurements can be viewed when selected.

Image files that show user interface displays of (a) a process class hierarchy for organizing related process models, which can be viewed when selected.

Image files that show user interface displays of (a) a meta-model class hierarchy we use, and (b) a meta-model schema for the "task-force" class can be viewed when selected.

A paper providing a more detailed description of meta-modeling can be found at