Careers: Interviews
Software Development Expert: Andy Carmichael

This week, Stephen Ibaraki, I.S.P., has an exclusive interview with, Dr. Andy Carmichael, an internationally renowned expert in software development methods and processes with more than 20 years experience in the industry. His most recent book, co-authored with noted J2EE expert Dan Haywood, “Better Software Faster,” is most definitely a ‘must’ read for all software architects, project managers and developers and the only book available that discusses their “single-source model” approach.

Discussion:
Q: Your list of accomplishments are impressive indeed! We thank you for sharing your years of experience and knowledge with us.
A: No need to stress the years. The grey hairs in the mirror are enough of a reminder!


Q: What do your partner and three daughters think about the attention you are receiving as a noted author, speaker, consultant, and internationally known guru?
A: Sharing my life with four wonderful young women is a privilege, though I don’t think it’s ever crossed their mind that I’m a guru! They are all involved in music and the arts and are only really interested in software when the computer they are using for school work, communication or recreation misbehaves. At this point software consultants are best advised to keep a low profile rather than boast of their skills!

Of course it should be a wake-up call to everyone involved in software when they come across users – especially ones they love – who are less than satisfied with the general quality of software! At least it encourages me to keep up the search for better ways to produce it.


Q: Can you describe what led you into the computing field and lessons you learned along the way?
A: Like many of my peers I didn’t start out studying software or computer science, but in using software to do something useful – in my case trying to calculate stress levels in aircraft components with holes in. This was a PhD research project in Mechanical Engineering at Southampton University in the UK. But soon afterwards I found myself working in a larger commercial team, where the problems of how programmers worked together, shared changes to a common code base and used tools to measure and improve quality, often outweighed the problems of how to program a particular algorithm or subroutine. This is when I first got involved in the evaluation and specification of software methods and then software development tools. Probably the most important lessons I learned in those years were the importance of the foundational principles of software engineering: abstraction, modularity, information hiding and dependency management (loose coupling and high cohesion). When object-oriented design and programming became a commercially viable prospect towards the end of the ’80s, it was clear to me that they added important mechanisms to improve the way we apply these principles, which was why I wanted to be involved in the exploitation of object-oriented methods.


Q: You founded Object UK Ltd. Can you tell us more about this company and its purchase by TogetherSoft?
A: I founded Object UK in 1992 in order to provide software tools and services to support object-oriented development methods. By then I had been working with CASE tools (Computer-Aided Software Engineering) for a number of years and I felt the emerging object-oriented methods would provide much needed impetus to this aspect of the industry. This proved the case, though the most significant development for Object UK was when the main product we were distributing, Together┬«, was rewritten in the new Java language and took the Java world by storm with its “single-source” approach. Basically it bypassed what had been the major Achilles heel of CASE products – the need to synchronize design and code through what is termed “forward and reverse engineering”. Not only is this difficult for tool vendors to implement, it’s incredibly difficult for developers to use effectively. Together overcame this weakness at a stroke by using the same source file – actual Java or C++ code – to store the models and diagrams that designers and developers drew and updated graphically. It was a master-stroke and one that even now the competition has not matched.

The main difference it made to us, as more and more development teams wanted to buy into the new approach Together had introduced, was that we found our whole business was now focused around this product. Since TogetherSoft wanted to have its own UK presence it was natural to begin talks to bring Object UK into the company that owned Together. We signed this deal in November 2000 and I then took on responsibility for growing the European team of mentors who would train and consult with the teams developing software with Together in Java, C++ and more recently C# and VB. Soon after that was also when Dan Haywood and I started work on the book Better Software Faster to provide guidance to development teams looking for new techniques to accelerate quality.


Q: You also edited two books for SIGS and Cambridge University Press—Object Development Methods (1994) and Developing Business Objects (1998). Can you share some tips from this experience?
A: Both these books were collaborations with other authors who were contributing to the commercialization of object-oriented approaches to software development. The early ’90s was a time when new object-oriented methods were being published almost every month! The situation was very confusing to project managers who just wanted to adopt best practice, rather than have to assimilate scores of different notations and processes then choose one or, possibly worse, invent a new one by combining different set of features from other published methods. The industry’s response to this situation eventually saw the emergence of a standard notation, UML (the Unified Modeling Language), first published in draft form in 1997 by Graby Booch, Jim Rumbaugh and others. In 1994 though my aim for the Object Development Methods book was that it would show the commonality of concepts and principles in all object-oriented methods, regardless of the notation people used to display them. I think this was reasonably successful and helped build consensus towards a standard approach. Developing Business Objects continued this work but including a wider scope of where object technology had been applied in business systems.


Q: Your most recent book, “Better Software Faster,” is so practical and it’s clearly evident that both you and Dan have considerable “real-world” experience to share. There is so much useful advice on how to engage and work on object-oriented, enterprise-based development projects. You use Together, TogetherSoft’s killer development app, and your sample distributed Java application is excellent—supporting both local and Internet-based clients. There’s an interesting foreword too by Edward Yourdon—he has been recommending this book to his friends apparently.

Will you talk about this book in general, what led you to write this book with Dan, and what you’re your main objectives with the book?
A: There were two clear goals we had when setting about this book. The first was to explain the “single-source model” approach and how this can improve quality and efficiency. This was not necessarily dependent on people using Together (providing they had an environment that would keep design and implementation in step). In many ways it’s much more important to me that people understand the power of this concept and implement it in other areas, rather than just go out and use Together. The lack of a single-source approach is, in my opinion, the most important reason why model-driven approaches and CASE tools in particular have not been used by even a majority of software developers. The second goal was to give Together users practical advice on how to get the best out of the product and guidance in defining their development process.

In fact we identified four principles that sum up the message of the book: maintain just one single-source model; use a minimum “meta-model” (in other words have a simple definition of the model behind your development process); use a “perturbation” change model (in other words an iterative approach based on small changes to a valid system); and continuously measure quality. These principles are applicable whatever development environment you use – or indeed if you are only planning to use a text editor and a compiler!

Dan and I had a good partnership on this book because while I’m always looking for these lofty goals and principles, Dan is the pragmatic details man and an excellent teacher of complex aspects of software. He produced an excellent Java implementation of our case study project – downloadable from the book’s web site (www.bettersoftwarefaster.com) so readers can use it with the book. He also provided a lot of detail, some in appendices to the book, on how to configure Together which have really been appreciated by Together users that I’ve spoken to. Dan’s deep knowledge of J2EE has also meant that we were able to cover this aspect of Java development in one of the later chapters of the book.

Ed Yourdon and I did some work together in 1992 but I had not seen him since, so I was flattered when he agreed to write the foreword to this book. Ed has done a lot of work recently on legal disputes between firms involved in large software projects that have gone wrong. As he points out in the foreword the very existence of these cases means that software engineering has a long way to go in improving its processes, so that the very common sight of large software projects ending in tears does not continue to happen. Ed picked out this paragraph from the first chapter of the book to highlight one key strategy for avoiding software disasters.

"By discovering what the business imperatives are, the team can deliver enhancements to the current capabilities within the specified period while also preparing for longer term requirements. If the team follows this route, it will need a flexible and lightweight development process with rapid feedback loops to report holdups and roadblocks—and it will need to plan contingencies accordingly."

Why read the rest of the book, asked Ed, if the answer to the problem is a straightforward as that?! Fortunately he found enough in the rest of the book to consider it essential reading. The fact that Ed is now recommending the book to business analysts, project managers, designers and developers who are not using Together, as well as to those that are, is a great encouragement to us.


Q: It’s often the case that the analysts’ detailed UML diagrams are overlooked or shelved by developers in the attempt to make deadlines. What’s the impact of such poor synchronization between the UML model and code and can you share you views on TogetherSoft’s software and how it addresses these kinds of issues?
A: You’re right Stephen, this is a common problem. When delivered software is out of step with the analysis and design documents there are two serious consequences. Firstly the information that has been carefully and expensively gathered in such documentation is seriously devalued, since the implementation is not consistent with it. This may result in incorrect functionality or delays caused by rework when users or testers discover inappropriate system behavior. Secondly it means that the higher level documentation is of only little value to the development team and subsequent maintainers of the system since, even if the documentation is consistent with the implementation, developers can’t assume it is, and they have to go and examine the source code any way – a much more time-consuming and error-prone process. Together’s LiveSource™ technology is the key to overcoming this problem. By actually deriving the higher level models from the source code and storing additional design documentation directly with source code files, there is no separate synchronization process to be carried out.

There is still the need of course for different levels of detail in the full set of diagrams and documents maintained by the development team. But rather than maintaining several models at different levels of abstraction, we advocate a “single-source” approach. This means that the different levels of class diagram for example are all derived from the same source files but with different filters and selections made on the diagrams by the designers, to ensure each diagram communicates with its intended readers. Diagrams produced to explain the detailed design for particular mechanisms in a system look very different from diagrams that show an overview of the components or the principle classes of business objects discovered in analysis. Yet all these diagrams can be derived from the same set version-managed files.

This approach works best with relatively small teams – that means more than a pair but less than a dozen developers – and one in which the different specialists, analysts, designers, database or user interface experts and programmers are all working together in the same team or teams, rather than separating out say analysis teams and coding teams. This makes sense in iterative life cycles where projects often have very short timescales (less than 9 months) and requirements and business priorities that may change through the life of the project. Such project require an “agile” process so that the team can respond to changing circumstances but still deliver quality software with useful and effective documentation.


Q: Can you comment on the iterative and incremental approach—the perturbation change model?
A: Sure. A perturbation is a small change away from a stable state – like moving a block by rolling it end over end. Each step in the process is a stable state and progress is made by temporarily “perturbing” the equilibrium. In an iterative development process it is important that the starting point for change is a stable point. In other words all the tests that have been defined for requirements implemented up to that point should pass, all the unit tests should pass and other quality measures that have been defined (such as measured complexity and coding standards) are also acceptable. Then when a change is made to the system, say to implement a new requirement, to improve its performance or to improve the integrity of its design, the quality measures can be reapplied and the system be brought back to its consistent working state. This contrasts greatly with a development process that separates out requirements definition, design, coding and testing into separate phases of the project.

Most modern software development processes are compatible with this perturbation model, but they also tend to define a lot of other aspects like project phases and roles which can cloud the issue. That’s why we’ve tried to explain development in terms of a simple continuous process that remains the same throughout the whole project: start from a state where the system (or model) is conformant with the quality measures, introduce a small change and get it back to the consistent state as soon as possible.


Q: Does this affect the way the project is managed as well as what the developers do?
A: It certainly does. We devoted one chapter of the book to project planning and monitoring because we think that project managers should be using a different planning model from the normal critical path analysis method, which is used by tools like Microsoft Project and is typically shown as “Gantt” or “Pert” charts. This method is essential to planning something like a building or a bridge where the dependencies between the tasks are extremely significant. You can’t put the roof on the house for example until you’ve built the walls to hold it up! On the other hand the order in which features of a software system are delivered to users are not constrained to the same degree, meaning that the users or business representatives can determine the priorities for the implementation of different features based on an assessment of risk and benefit. We can also allow these priorities to be changed as the project progresses. This makes the development plan flexible, while still enabling close monitoring of progress and continuously adjusted estimates of completion dates.


Q: Sounds as though you’ve had experience of projects that don’t always deliver to plan. Have some of the projects that you have worked on had problems like these and what lessons were learned by the project that you can pass on?
A: You’re right Stephen! I don’t think anyone who has worked in this industry for any length of time would not have seen some project or other suffering from budget or schedule overruns. I remember one large project where everyone was working really hard to meet its first customer delivery deadline – not for the system going live but still a very important milestone. Right up to a week before the deadline the project manager thought he was going to get a system out on time. However at that point they realized that it was not going to happen and they declared a one month slip. How could the management have been so out of touch with the reality on the ground? Well in fact the project did not complete that milestone until a full 5 months later! The main problem was that this was the first occasion when the project had tried to integrate all the software being delivered by the different teams. Everyone thought they could finish their piece in time but it wasn’t until they started to put them together that they could see where the holes were. Continuous integration of software, and continuous unit and functional testing of software, have been important aspects of software development processes that I stress to my clients, and particularly since the experience on that project. Whether the project learned the lesson I’m not sure, but certainly many other projects have learned from their problems.


Q: I understand that you’re now moving on from TogetherSoft? What does the future hold for you now do you think?
A: That’s right. I’ve really enjoyed my time working at TogetherSoft with a tremendously talented bunch of people, both in the UK and throughout their offices worldwide, but it is now time to move on. I intend to remain available as a consultant to clients using Together as their development platform but I also want to consult more widely with enterprises defining and improving their development processes. I would not be surprised to find that I become involved again in the management of a UK-based professional services organization but actually at this stage I haven’t made any final decisions about new businesses. It’s one of the bonuses of life that one never knows what’s just around the corner!


Q: Thank you for sharing your valuable insights with us today and we look forward to hearing you speak at conferences, reading your books, and articles.
A: Thank you Stephen. It’s been a pleasure.


Copyright Network Professional Association® 1994-2017. All Rights Reserved.
NPA Privacy Statement