Our Changing World of Software Industry

Scientific requirements engineering is a discipline that is based on three theoretical foundations: systems thinking, mature fundamentals of design, and the methodology of deductive science.

Systems thinking reconceptualize enterprise software and requirements. The high waste resulting from failed projects in the software industry, especially that associated with large-scale systems failures, indicates that the system of beliefs that supports thoughts about systems design is grossly underdeveloped and underconceptualized. These underconceptualized definitions and models are the direct results of the assumptions held by the discipline. Those assumptions largely determine what the practitioners assume to be reality and “facts,” establish what to focus on, and indeed determine what the discipline is all about. The transition from a paradigm in crisis to a new one is a reconstruction of assumptions and resulting methods. When the transition is complete, the profession will have changed its view of the field, its methods, and its goals. When paradigms change, the world itself changes with them. Systems thinking offers the new concepts from which the new paradigm can be easily conceptualized.

Mature fundamentals of design common to all branches of engineering prove to be an effective starting point for a field in its immaturity. Once the mature fundamentals are found, the new discipline of software engineering that complies with these fundamentals of engineering design can be constructed. This in turn will solve the problem of theory being decoupled from practice in current practice of software engineering. For any branch of established engineering, the main task of engineers is to apply their scientific knowledge, the knowledge of the design constituents and their governing laws, to their technological knowledge, the knowledge of design, to implement required functions and to solve technical problems. For enterprise software, the scientific knowledge is a business model that has to be created real time.

The methodology of deductive science is the methodology to create scientific disciplines such as mathematics. Different from today’s civil engineers whose scientific knowledge (natural sciences) is given and taught in their school years, software engineers would have to create organizational science (business model) on their own each time they work on a project. For anyone who intends to study or advance some science, it is undoubtedly important to be conscious of the methodology employed in the construction of that science, and we shall see that, in the case of organizational science, the knowledge of that methodology is of particular far-reaching importance, for lacking such knowledge makes it impossible to comprehend the nature of social organizations. The principles with which we shall get acquainted in the methodology serve the purpose of securing the knowledge acquired in the business model of the enterprise software at the highest possible degree of clarity and certainty. From this point of view, a systematic methodology of structuring and visualizing the enterprise software requirements is both necessary and sufficient to revolutionize the software engineering from guesswork to scientific work. This methodology is the realization of the methodology of deductive science in the context of enterprise software.

Traditionally, software development processes have been requirements-driven: an attempt is made to provide requirements definitions and then to implement those requirements. Therefore, the success or failure of requirements determines the success or failure of software projects. In real world however, virtually every major software program suffers from severe difficulties in requirement specification.  Many studies cite requirement management issues as a primary cause of software project failure.

Requirements are typically defined as what the system should do, as explained in Wikipedia:  “In systems engineering, a requirement can be a description of what a system must do, referred to as a functional requirement. This type of requirements specifies something that the delivered system must be able to do. Another type of requirements specifies something about the system itself, and how well it performs its functions. Such requirements are often called nonfunctional requirements, or ‘quality of service requirements.’ Examples of such requirements include availability, testability, maintainability, and ease of use.”

Professional developers usually develop software for someone other than themselves – they build it for the users of the software. It is assumed that the users must know what they require. However, little experience trying to gather requirements from users soon reveals them to be an imperfect source of information. Frankly, users frequently do not know what the requirements are or how to specify them in a precise manner either. Even with the help of analysts, user did not fully understand what the software system ought to do. As projects proceeded, users and the developers themselves could see what the system would look like and thus came to understand the real needs better, a wealth of change would be suggested.

The source of functional requirements is the opinions of the users and developers. They are ambiguous and largely depend on the individuals’ experiences and preferences that vary from time to time and from people to people. Hence the software system is built on sand (personal opinions) instead of rock (stable scientific principles). This shaky foundation on which the software system is built is the root cause of the problem.

The functional requirements, what the system should do, by definition is the solution. There is no scientific knowledge, the law of “nature” to be applied in creating the technical knowledge. The fundamentals of design common to all engineering is a process of applying scientific knowledge to create technical knowledge by which the design constituents are harnessed and intended utility or functions are implemented. Therefore the problem of requirements engineering cannot be solved for the lack of scientific knowledge.

Fred Brooks drew on an ancient philosophical tradition of distinguishing between “essential” and “accidental” properties to describe software. Essential properties are those properties that a thing must have to be that thing. A car must have an engine, wheels and a transmission in order to be a car. A car might or might not have a V-8 or V-6 engine or an automatic or manual transmission. These are “accidental” properties that do not affect the basic “car-ness” of a car. Software entity, within the context of enterprise, is described in two aspects: essence inherent in the nature of software and accidents attending its production but not inherent.

The essence of a software entity is the scientific knowledge that is a business concept: a construct of business processes, business rules, entities, relationships, algorithms, and invocations of business functions. This business concept is abstract in that such a conceptual construct is the same in view of different software technologies. It is nonetheless highly precise and quantitatively detailed and hence the “essential” property. The accident of a software entity is the technical knowledge that represents the essence by means of computer languages in the form of graphic user interfaces, databases, components, middleware, and network protocols, etc. This representation is concrete in that such representation can be in many different ways subjective to IT workers’ experience and preferences. It is nonetheless highly diversified and qualitative and hence the “accidental” property.  Creating technical knowledge without creating and applying scientific knowledge first is the root problem of requirements engineering.

The immaturity of software engineering discipline can also be seen from the revolution cycle of scientific disciplines and their formation. According to Thomas Kuhn, scientific revolutions come about because existing paradigms no longer solve existing problems, creating “anomalies” that lead to a crisis. Scientists then start to scrutinize the current paradigm itself and start coming up with alternative paradigms. Eventually, a new paradigm that has the power to resolve the anomalies establishes itself. When a new paradigm is finally adopted, science will have undergone what Kuhn calls a scientific revolution. After the revolution, a new normal science is stabilized. Kuhn’s revolution cycle starts from a pre-paradigmatic stage described below.

The pre-paradigmatic phase represents the “pre-history” of a science, the period in which there is wide disagreement among researchers or groups of researchers about fundamental issues, such as which phenomena should be explored and according to which theoretical principle; what are the relations of the theoretical principles with each other and with theories in related domains; what methods and values should guide the search for new phenomena and new principles; what techniques and instruments can be used, and so forth. While such a state of affairs persists, the discipline cannot be said to be truly scientific. A discipline becomes scientific when it acquires a scientific paradigm, capable of putting an end to the broad disagreement characterizing its initial period. At this stage, the discipline becomes a science. Within the new paradigm, the discipline sets the problems, the terms in which these may be approached to give a valid solution and the means of identifying what constitutes a valid solution. It presents challenging puzzles, supplies clues to solutions and guarantees the competent practitioners success that those of the prescience schools did not. This activity of puzzle-solving within the constraints of the paradigm is referred to by Kuhn as normal science.

Software engineering can safely be considered to be in crisis within its current paradigm for wide disagreements on the list of problems. The theoretical foundation of Unified Process no longer meets the demand of today’s complex problems. There are several indicators that point in that direction, like huge diversity of development methodologies etc., and wide disagreement among researchers and practitioners about its “scientific paradigm” (i.e., its formal theoretical foundations). The recognition of current status of software engineering being a Kuhnian crisis gives us a clear understanding of where software engineering should be heading and what should be done about the current crisis, the emergence of a new paradigm to put an end to the methodology war. The decision to reject one paradigm is always simultaneously the decision to accept another, and the judgment leading to that decision involves the comparison of both paradigms with nature and with each other. The transition from a paradigm in crisis to a new one from which a new tradition of normal science can emerge is far from a cumulative process, one achieved by an articulation or extension of the old paradigm. Rather, it is a reconstruction of the field from new fundamentals, a reconstruction that changes some of the field’s most elementary theoretical generalizations as well as many of its paradigm methods and applications. When the transition is complete, the profession will have changed its view of the field, its methods, and its goals.

The mature software engineering discipline lies in its alignment with the mature fundamentals of design common to all branches of engineering. It must include the science at the lower level for the principle of design to be scientific. This science is not already given like mathematics but must be created real time. It is the organizational science in the application domain, called business model. It is applied science because it is derived from organizational and systems theories. It models relative unchanging business aspects such as the types of customers, services and products, and value paid for.

Engineering is the activity of designing a machine or a factory.  Operating a machine or running a factory is not the act of doing engineering.  Design is the act of doing engineering and it is a creative act.  Dictionary definition of engineering is “the application of science and mathematics by which properties of matter and the sources of energy in nature are made useful to man.” This definition talks about two levels of knowledge: scientific (science and mathematics) and technological (making materials into devices useful to man) knowledge. Scientific knowledge is universal and invariant and talks about what the world is. Technological knowledge is contextual and volatile and talks about what the world should be.

Dictionary definition of machine is “an apparatus for applying mechanical power, consisting of a number of interrelated parts, each having a definite function.” The manufacture of a machine consists in cutting suitably shaped parts and fitting them together so that their joint mechanical action should provide required function. The structure of machines and the working of their structure are thus shaped by man, even while their material and the forces that operate them obey the laws of inanimate nature. In constructing a machine and supplying it with power, we harness the laws of nature at work in its material and in its driving force and make them serve our purpose. This harness is not unbreakable; the structure of the machine, and thus its working, can break down. But this will not affect the forces of inanimate nature on which the operation of the machine relied; it merely releases them from the restriction the machine imposed on them before it broke down.

The machine, as a whole, works under the control of two distinct principles. The higher one is the principle of the machine’s design that harnesses the lower one, which consists in the physical-chemical processes on which the machine relies. The lower one is the foundation where isolated particulars leave indeterminate conditions to be controlled by a higher principle. The higher one is a result of creative activity from which new properties emerge by ways of structuring constitutes of the level below. Upper level properties are emergent in sense that they are not reducible to the lower level principles. The shape of a cup is not reducible to the law of physics. The lower level is the foundation and is independent of the level above. The higher level depends on the level below for its implementation.

Therefore, the fundamentals or the essence of engineering design is a process of creating scientific knowledge (if not already provided) at lower level and then transforms it into technological knowledge at the higher level. The main task of engineers is to apply their scientific knowledge to create their technological knowledge by which the constituents are harnessed and intended utility or functions are implemented. They then optimize these solutions within the requirements and constraints of the project. For example, electronic engineers apply the knowledge of electronic properties of engineering materials to design circuits where the electronic properties of the materials are harnessed or channeled to perform the desired function.

Scientific knowledge is the human’s discovery of “law of nature.” There is definite regularity or order underlying nature for men to discover. Without a consciousness of regularity the idea of natural law is impossible. Indeed, there can be no science without the concept of order in nature, for it is this very order which science seeks. It is the invariant nature of natural laws that gives the principle of design the stability, sustainability and operability. The scientific knowledge and its useful application is the source of theory the practices of a field rely on. Without theory there is no learning. Without theory, there is nothing to revise and nothing to build on. Theory based on science predicts results and anticipates success. Engineering activity without theory, the understanding of law of nature and its useful application, is guesswork rather than scientific work. Guesswork is guided by opinions instead of theory. Practice being decoupled from theory is manifested in all immature fields where opinion based trial and error dominates. The design of a structure or a mechanical device to carry maximum loads or perform a specific function, for instance, is the most precise and economical when scientifically designed while imprecise and wasteful when designed on the basis of “experience.”

Software industry is about half century old and has been pictured with excessive schedule pressure, long overtime, constant change and frequent overruns. This is because software engineering as a new engineering discipline is still developing hence immature. This can be evidenced in view of the larger historical context of engineering. From this view, we can see not only where a developing discipline is in its course of development but more importantly how should we go about to achieve its maturity.

The history of engineering can be better understood by looking at the relationship between science, an abstract interest in nature, and technology, the crafts or tools. In most human history, science and technology remained the largely separate enterprises, intellectually and sociologically. They had been since antiquity. The technology of the Industrial Revolution remained in classical independence of the world of science; only during the nineteenth and twentieth centuries did thinkers and toolmakers finally forge a common culture. Historically technology helped science more than science to technology. It frequently happened that technology has made instruments that advanced scientific discoveries and discovered new methods that challenge scientists for new theories to explain what happened. Telescope, for example, led to better astronomical measurement, which led the idea of heliocentric view.

Engineering without science is imprecise and uneconomical. The Romans appear to have had no theory regarding of stress, thrusts, and distribution of weight. There was no science of statics until after 1500 in the Renaissance. Roman engineers made no quantitative tests or the strength of materials under tension or compression or bending or shearing. They did not realize that the strength of a beam depends upon the shape as well as upon the area of its cross section. They built their huge aqueducts and bridges solidly with caution and common sense, well within the appropriate factor of safety or margin of error. At the same time, however, the Romans were deliberately raising too perilous heights apartment houses that frequently collapsed.

Science and industry and the cultures of science and technology generally began their historical merger in the nineteenth century. Since then scientific knowledge has been applied to improve and even predict technologies. The flow of information has been from science to technology. Radio communication is a good example where practical application followed closely on the heels of theoretical innovation. The flow of scientific information to complex structures leads to lasting and economical bridges and high-rise buildings. An engineering discipline becomes mature only when science and technology are fully merged and a single methodology based on standards is created. Personal opinions are ruled out for placement. When the standards are followed, bridges will be on schedule, on budget, on spec and do not fall down.

In the software industry, software engineers are much like civil engineers before industrial revolution Both of them build bridges/software by tingling hence most of them fall down. Both have myriad methods and there was no standard to mandate how to build bridges and software. After industrial revolution, civil engineers would be able to specify requirements in terms of scientific principles, resulting in precise, concise, and stable requirements. There has been continuous progress in information technology such as computer languages, integrated development environments, and network protocols. But in terms of progress in scoping and representing problem space, there has been none. There is no science used in the software industry up to date to specify precise, concise and stable software requirements in the same way building requirements are specified in terms of physics and mathematics. Because software engineers have more common with craft men than scientists, therefore software engineering is an immature field.