In order to compete successfully in the open market software companies need to improve both the quality of software products and ability to work within time and budget constraints. These improvements are based primarily on process and only secondary on technology. As software engineering is one of the pillars of the emerging information society, it is obvious why the efforts in the software improvement are encouraged on global basis.

This paper is intended to present such software process improvement efforts and their effectiveness. We begin with the summary of software process improvement models in use (CMM, ami, Bootstrap, ISO 15504 SPICE and ISO 9000). Then we present results of software process improvement in the world and Croatia.

Here we shall present some of the most important software process improvement models: SEI's CMM, ISO 15504 SPICE, ami, Bootstrap and ISO 9000.

2.1 SEI's Software Capability Maturity Model

The SEI's Software Capability Maturity Model (CMM) is the first software maturity model, and is developed to describe the principles and practices underlying software process maturity. Its aim is to help software producing organizations improve the maturity of their software engineering processes. It uses the paradigm of evolutionary path - from ad hoc processes to mature and disciplined software engineering. The CMM is organized into five maturity levels.

The first level is called Initial. The software process can be characterized as ad hoc and chaotic. Processes are practically not defined, and everything depends on individual effort and capability.

The second level is called Repeatable. There are basic project management processes to manage cost, schedule, and functionality. Software processes are basically disciplined in order to repeat earlier successes on similar projects. The key process areas at Level 2 are: Requirements Management, Software Project Planning, Software Project Tracking and Oversight, Software Subcontract Management, Software Quality Assurance, and Software Configuration Management.

The third level is called Defined. The software process can be characterized primarily as documented, standardized, and integrated regarding management and engineering activities. The key process areas at Level 3 are: Organization Process Focus, Organization Process Definition, Training Program, Integrated Software Management, Software Product Engineering, Intergroup Coordination, and Peer Reviews.

The fourth level is called Managed. It is characterized by collecting detailed measures of the software process and product quality. The software process (and products) are quantitatively controlled. The key process areas at Level 4 are: Quantitative Process Management and Software Quality Management.

The fifth and the highest software engineering maturity level is Optimizing level. Quantitative feedback from the process is used to enable continuous process improvement. The key process areas at Level 5 are: Defect Prevention, Technology Change Management, and Process Change Management.

Overall maturity of an organization's software engineering processes is being improved as the organization moves up the scale of five maturity levels. This is strongly supported by the empirical evidence to date.

2.2 ami

ami is defined as a quantitative approach to software project management and process improvement. Its proclaimed goal is to make the best software measurement technology available to all the software engineering staff. ami integrates the concepts of software process maturity assessment and goal-oriented measurement into a twelve-step method aiming at the organization business goals.

Without a structured method, it is not possible to apply the goal-oriented measurement and build a coherent measurement framework. As a result, the organization can spend much time chaotically trying out many methods, but before a method can be built into the process, it is usually abandoned.

The ami method implements four activities - assess, analyze, metricate and improve - in an improvement cycle. The final improvement step can lead to the identification of further improvements. These four activities are:

- ASSESS your project environment to define primary goals for measurement.

- ANALYZE primary goals to derive sub-goals and identify relevant measurements.

- METRICATE by implementing a measurement plan.

- IMPROVE by using the measurement data to drive software process improvement decisions.

2.3 Bootstrap

The Bootstrap method was developed under European development program ESPRIT. Republic of Croatia has accreditation for its use through the Croatian Information Technology Agency. The assessment procedure is presented in Figure below.

Assessment preparation consists of opening management briefings, method explanations and its usage, expected results and post-assessment improvement actions.

Method provides measurement of the whole organization maturity level (SPU in Bootstrap terminology) and/or a particular project. SPU assessment includes measurement of about 130 state indicators through the work reviews, documentation reviews and conversation with senior management as well as with project managers. Project measurement is conducted in a similar way by measurement of about 110 indicators through the project documentation reviews and conversation with team members.

These measurements provide data for maturity level calculation. Algorithm calculates total maturity level of the SPU/project, and also maturity levels of 18 key areas.

2.4 ISO 9000

ISO 9000 is a series of quality assurance standards developed by the International Organization of Standardization that gives an overview, defines terminology and provides guidance for selecting the other standards in the series (ISO 9001, 9002, 9003, and 9004). As its part ISO 9000-3 is specially concerned with applying quality principles to software development. ISO 9001 is the most comprehensive standard that provides guidance to companies on how they should organize their quality management system. The improvement of software process can be attained through good quality system and program. At the heart of the application of quality program there is something called a quality system or, as it is increasingly known, a quality management system. This consists of the managerial structure, responsibilities, activities, capabilities and resources that ensure that software products produced by projects will have the desired quality factors that both the customer and the developer decide will be built into them.

The specific details of a quality management system will be held in a quality manual. Such a manual will normally contain standards for the quality and developmental activities that may be applied to a project together with detailed descriptions of quality controls. The requirements of the standard are partitioned into 20 headings:

- Management responsibility - Quality system - Contract review - Design control - Document control - Purchasing - Purchaser supplied product - Product identification and traceability - Process control - Inspection and testing

- Inspection, measuring and test equipment - Inspection and test status - Control of nonconforming product - Corrective action - Handling, storage, packaging and delivery - Quality records - Internal quality audits - Training - Servicing - Statistical techniques

2.5 SPICE (ISO 15504)

SPICE (ISO 15504, since the end of 1996) is a major international initiative to develop a Standard for Software Process Assessment and Improvement. The project SPICE (Software Process Improvement and Capability dEtermination) is carried out under the auspices of the International Committee on Software Engineering Standards ISO/IEC JTC 1/SC 7, through its Working Group on Software Process Assessment (WG 10).

Through SPICE, International Standards Organization (ISO) has an intention of developing a framework standard for software process assessment, bringing together the major suppliers and users of assessment methods.

Main parts of the SPICE version 2.00 (October 1996.) are:

- Part 1 : Concepts and introductory guide

- Part 2 : A reference model for processes and process capability

- Part 3 : Performing an assessment

- Part 4 : Guide to performing assessment

- Part 5 : An assessment model and indicator guidance

- Part 6 : Guide to qualification of assessors

- Part 7 : Guide for use in process improvement

- Part 8 : Guide for use in determining supplier process capability

- Part 9 : Vocabulary

3.1 Application and measurement results of the Bootstrap method in Croatia

Using the Bootstrap compatible method, 15 assessments have been conducted in Croatia so far. Three of these were SPU assessments and the rest of them were project assessments. In three cases SPUs were part of the larger business system while other companies were independent software producers.

Some of the results are shown in the Table 1.

	Maturity Level
	Max	Min	x	s
Total Maturity Level	2.75	1.25	1.85	0.46
Organization	3.00	1.50	2.00	0.44
Coordination	3.00	1.50	2.38	0.41
Development	2.25	1.50	1.71	0.29
Validation & Verification	2.00	1.50	1.71	0.25
Project Support	2.00	1.25	1.56	0.29
Life Cycle Independent Functions	2.50	1.25	1.83	0.39
Configuration Management	2.75	1.25	2.33	0.19
Risk Management	2.50	1.00	1.48	0.51
Project Management	2.75	1.25	1.6	0.044
Quality Control	2.50	1.00	1.79	0.43
Subcontractor Management	3.00	2.25	2.17	0.72
Process Related Functions	3.00	2.00	2.31	0.36
Process Description	3.00	1.50	1.94	0.49
Process Measurement	3.50	2.50	2.94	0.38
Process Control	3.00	1.00	2.23	0.58
Life Cycle Functions	2.75	1.75	2.19	0.33
Development Model	3.00	1.00	2.37	0.49
Requirements Specification	2.50	1.50	2.06	0.42
Architectural Design	3.00	1.00	1.77	0.65
Detailed Design	3.50	1.75	2.52	0.73
Testing	2.50	1.75	2.00	0.20
Methodology	2.75	1.50	2.06	0.33

On the basis of these measurements, it is estimated that the expected maturity levels of the Croatian software companies are between 1.25 and 2.75, with the average value of 1.75. Project development organization is rated between the 1.50 and 3.00 (expected 2.00), and methodology between 1.50 and 2.75 (expected 2.00).

In the organizational domain, the weakest points are development organization, verification and validation, and project development support (ML=1.50). On the opposite, coordination during life cycle is one of the best performed functions in the software companies (ML=2.25).

In the methodology domain, process dependent and life cycle functions (2.25) are better side of the process regarding the life cycle independent functions (1.75).

Observing life cycle independent functions, we can see that configuration management (2.25) and subcontractor management (2.25) are better performed than risk management (1.50), project management (1.50) and quality control (1.75).

Observing life cycle functions, we can see that requirements specification, development model, architectural design and testing (1.75-2.25) are weaker than detailed design (2.50).

Regarding process functions, one can see that process measurement is better defined and practiced than process description (2.00) and process control (2.25).

Technology measurement (development tools) has shown that state-of-the-art technologies are not being used throughout process (they cover only 30%). Tools are not used at all in requirements specification (0%), or architectural design (10%) contrasted to detailed design an implementation (70%). Also, project management tools usage is very low (5-10%), just as risk management (0%), testing and integration (10%), etc.

3.1.1 Results interpretation

Expected maturity level of the average Croatian software company is 1.75. This result shows that Croatian software industry is stepping into the phase of so called repeatable production. There are characteristics of both the industrial and the craft production. Production process is not defined, methods are poorly used and individual talent dominates in the production (mainly project managers and key members of the development team). Tool usage skill still has an advantage compared to usage of mature methodologies. Products are more unique, and still lack quality of industrial product. However, it is noticeable that there is common view that this situation should change. Some actions are taken in this direction, but they are still particular and instinctive.

Difference between the lowest (1.25) and highest maturity level (2.75) indicates large diversities on the market (from reliable to highly risky companies). Differences in observed sample depending on the company size and ownership have also been noticed. Market driven companies are more interested in acquiring new knowledge and technology transfer, but they are mostly understaffed. Most of the companies have development teams with only 2 to 4 team members, and unexpected departure of one of them may cause serious problems in project development - what makes these companies more unstable.

In the organizational domain, coordination is stronger than development organization, validation and development support. That is because SPU project managers, usually very experienced and educated stuff, correctly plan and manage resources and their responsibilities. But validation and verification, as well as the configuration control functions are not supported enough.

In the methodology domain it is interesting that its maturity is equal to organization. Focus of attention was directed towards process related functions - process measurement and control.

Expected Croatian software producers maturity level and its characteristics are shown in table 2.

Total Maturity Level	1.75		Process Related Functions	2.25
Organization	2.00		Process Description	2.00
Coordination	2.25		Process Measurement	3.00
Development	1.75		Process Control	2.25
Validation&Verification	1.75
Project Support	1.50
Life Cycle Independent Functions	1.75		Life Cycle Functions	2.25
Configuration Management	2.25		Development Model	2.25
Risk Management	1.50		Requirement Specification	2.00
Project Management	1.50		Architectural Design	1.75
Quality Control	1.75		Detailed Design	2.50
Supplier Management	2.25		Testing	2.00
			Methodology	2.00

3.2 The surveys of software process improvements using ISO 9000

(Dirk Stelzer, Werner Mellis, Georg Herzwurm, University of Cologne, Software Process Improvement via ISO 9000?)

Do the ISO 9000 standards have influence upon software process improvement? The staff of many companies ask and think about that. Some surveys of this problem have been conducted in Europe.

University of Cologne has conducted two surveys among European software houses. The first one was carried out in Germany. It involved twenty software houses that had an ISO 9001 certificate. They checked five elements of an ISO 9000 quality system: code reviews and inspections, software testing, product and process measurements, measurement of quality costs, and demonstration of quality improvements. Most software houses do not have important improvemnts in these areas. In despite of that, all of these companies would implement ISO 9000 quality system again.

The second survey was conducted among 36 European software houses. The survey has proved that among these software houses SPI via ISO 9000 were realy substantial but badly quantified.

University of Cologne has identified ten key success factors that are very important at implementation of ISO 9000 quality system. Since only two of these factors are direct requirements of ISO 9001, it is obvious that ISO 9000 QS implementation has to be a more comprehensive procedure.

3.3 The survey by ESPITI

(Helmut Rempp, ESPITI survey reveals scepticism about ISO 9000 and training)

The ESPITI consortium (European Software Process Improvement Training Initiative) carried out the Europe-wide user survey of the software industry in 17 countries. They came up with the following results:

1. The awareness level of SPI is low in all industrial sectors.

2. The major problems in software development are:

- requirements specification
- managing customer requirements
- documentation
- software and systems testing
- lack of quality systems, standards, structured approach
- project management.

That is presented in Figure below.

3. The commercial managers who decide about investing in software process improvement are not so informed as technical managers and staff. Nevertheless, 35 % of the respondents are using SPI tools and techniques, and 20 % are planning to use them. Only 16 % think that SPI does not have relevance to their needs.

4. The significant part of European managers and their staff cannot recognize the importance of ISO 9000 quality standards concerning software development. So 25 % of the respondents do not believe that ISO 9000 is relevant to their software development processes and 33 % do not believe that other assessment methods are relevant. In despite of that, 40 % of all respondents are using or planning to use ISO 9000 for support and 30 % are using or planning to use other models.

5. ESPITI has recognize the importance of training through more forms. One of them would be information seminars that are subject of interest for 39 % of the companies. More than 20 % are interested in training in working groups, in which companies would be able to discuss or work on common problems.

6. The part of survey that concerns training revealed that 32 % of the companies are interested in special training courses. 90 % of them want that these courses last not longer than three days.

7. The training budgets of the companies are very low. 30 % of the respondents are planning only three days' training per employee per year in the software area.

3.4 Results of improvement activities using CMM

The newest results of improvement activities using CMM (Process Maturity Profile of the Software Community 1996 Update - SEMA.11.96, September 1996) show:

- Number of organizations initiating software process improvement continues to increase.

- Increasing proportion of commercial and in-house organizations.

- Manufacturing organizations are conducting the most software process assessments.

- Nearly half of the organizations reporting size have less than 100 software personnel.

- Overall community profile continues to shift towards higher maturity.

- USA and non-USA profiles virtually the same.

- The KPA profiles from CBA IPIs for organizations assessed at level 1 indicate that Software Quality Assurance, Software Project Planning and Software Project Tracking and Oversight are the least frequently satisfied KPAs.

- The KPA profiles for organizations assessed at level 2 indicate that Integrated Software Management, Organization Process Definition and Training Program are the least frequently satisfied KPAs.

- Higher maturity reached among those organizations reporting reassessments.

- Relatively low number of reassessments reported compared to number expected: approximately 22%.

- Based on reported reassessments, median time to move from ML 1 to ML 2 is 35 months for all organizations and 26 months for organizations that began their CMM- based SPI effort in 1992 or later.

- For all groupings similar pattern for ML 1 to 2 and 2 to 3 movements: ML 2 to 3 tends to be faster and have less variance.

Presented data is outcome of 645 assessments conducted in 542 organizations (94 organizations have been reassessed) on 2812 projects. 19% of the assessed organizations are organizations outside of US.

On the basis of results presenting effects of software process improvement efforts we can observe continuous increase of software development maturity. Large number of organizations with very different software products have approached software process improvement in a variety of ways within the context of CMM, Bootstrap, ami and other models. More and more assessments are being carried out during last few years, and this number is constantly increasing. Software companies are getting more aware of the fact that improvement of software development process can provide long-term benefits.

These results can be explained with the increase of level of critical knowledge and the increase of discipline in software development - which represents a step further in attempt to move software development from the craft approach to maturity of industrial production. And as being a brick of critical importance in building the information society, every software manufacturer is called upon to accept new knowledges.

Domagoj VUCICA, B.Sc.
Consultant
PULSAR
Dubrovacka 57
21000 Split

Silvana TOMIC ROTIM, B.Sc.
Project manager
CROATIAN INFORMATION TECHNOLOGY AGENCY
Mazuranicev trg 8
10000 Zagreb