Fabian Schonholz’s Blog

Software Development Methodologies have been at the core for solving problems with computer software. There are many reasons why a methodology is important, but mostly, because it gives projects a beginning, a middle and an end. Projects are like stories and as such, they can have good or bad endings.

Let’s define what a Software Development Methodology is (I will be broad to serve the purpose of this post):

“A Development Methodology is a repeatable series of steps followed in order to complete a project involving the development of computer programs.”

Development methodologies are particularly important when there is a team working on a project. If just one person is working on the project, then the methodology is simple: just crank the code out. But when more than one person are working together then the level of complexity also increases proportional to the number of individuals involved in the project. Additionally, after the number reaches critical mass (defined as the absolute number of people with the right skill sets assigned to the project), one more person increases the complexity in an exponential manner. This WILL derail the project and the more people are added, the worse the project will fair. The proverbial “Mythical Man Month”.

The way I like to organize teams is as follows:

• Developers: Lead by a development manager, this group is responsible for coding. There can be more than one development team working on different components, products, releases, etc.
• Integration and Release: Lead also by a manager, this team is responsible for engineering the release during its different stages and for completing the release altogether.
• QA: lead by a QA team leader, this group is responsible for the engineering of tools used for automating regression and basic use cases.
• User Acceptance Testers (UAT): Also led by a team leader, they are responsible for functional testing. In other words, they look at the functional spec and press buttons to verify the application works as expected.

I left out two very important roles, project management and architecture.

I am normally heavily involved in architecture. I have some very definite ideas on how to architect massive distributed systems and I am not yet willing to relinquish that responsibility, but when I do, architecture and design are more general management areas than production. They do ensure that the product is built right, but that is true across all products being built. Also, the responsibilities of an architect extent pass the boundaries of projects into integration with outside products and services.

Project management, even though it is an incredibly important function, I see it as external to development as well. Indeed, it does set the mood, pace and organization of development, but still, I feel very strongly that it is more a part of management and administration than development.

Incidentally, I have the best Project Manager working for me: Sue Christensen. She rocks. She is the best and to boot, she complements me very well, specially when we do not see eye to eye.

Similarly to the way I like to set teams up, I like to set up operating environments in the following manner:

• Development: This represents each one of the developers’ workstations. Each developer should have a scaled down version of the application. The code is kept locally for the purpose of development, but it MUST be checked into a code repository like CVS or SVN on a regular basis, provided the code compiles.
• QA: This is a separate environment where the code is deployed for load testing, regression testing and other automated tests. There are multiples of these environments so different releases or components can be tested simultaneously. These multiple environments are key to the pipeline.
• Staging: This environment is a facsimile of production but scaled down. It needs to accurately represent production configuration so similar problems are manifested and they can be addressed before the release goes live.
• Production or Live: This environment is SACRED. It is the environment that is used to operate the company.

The Pipeline

I see software development as a manufacturing process. Do not be mistaken though, I am very much into innovation and I see it core to my functions and what, as a CTO, I bring to the party; but innovation should happen outside of production and brought into production as part of a product rollout. In a sense, it is similar to architecture, design and project management. They are all external to the actual production and manufacturing of a technology product.

But what is the pipeline?

The concept of a pipeline is to carry out no less than a release per week. Ideally releases would happen every day, except for Fridays, but definitely not less than one. Each of these releases can be a bug fix, enhancements or even new products. But for each release to be put into the release pipeline a process needs to be followed.

This process is simple. Let’s take a new product for example. The product development group is responsible for writing the business, which should explore and document the business logic for the given product. In a perfect world the business requirements will be closed and complete, which is a pretty way to say there should be no holes in the document. This document is given to the project management group for review.

The project management group develops the functional specifications, which are sent to the development group for review. Of course there are consultations between the product and the project managers as the functional specs are written. Once development has a chance to review, a Joint Application Development (JAD) session is conducted. All concerns are brought up at the JAD session and documents modified. After a quick get together and making sure all is good, development starts. In some organizations before development starts there is a very formal process where the documents are approved and signed-off. I think it is too formal a step and should be avoided.

Development should work towards quickly building a functional prototype. The prototype should be bare bones of what the final product will be, but it should start showcasing the beginnings of that product. As development proceeds new versions of the prototype are deployed into a QA environment for everybody to start playing with each new version. Each one of the prototypes cycles should take no more than a week. Sure, there will be some features that will take longer to develop, thus they should either get developed as part of the initial prototype; or in a different branch and merged into the release branch once ready.

There are some very tangible benefits to having these prototypes rolled out early:

1) Transparency. Everybody can see how the project is shaping up.
2) Expectation Management. If there is slippage, it will become apparent immediately. Also if a feature is less than expected, it will become apparent as well.
3) Market Deviations. If a product takes too long to develop the market it was going to serve might have changed. Early prototypes allow product managers and management in general to measure the developing product against the market and make the necessary changes without derailing the timelines too much.
4) Mood. Developers see their work materialize faster. They get happier and work faster.
5) QA. The QA team can start building and running automated test cases against the prototypes and provide early feedback to developers, minimizing the QA and bug fix time at the end of the project.
6) UAT. The UAT team can also start setting up the right cases and start exercising them so as to get readier when the time to execute comes. Also, UAT can double up as a documentation team during the early stages.

As the process moves forward more and more of the product is showcased through the prototype cycles, until one day it is ready for the final push and QA, UAT and development get the product into the pipeline and finally out the door.

In what I outlined above there is nothing new. Except, if you have noticed, there is an active role for everybody. Nobody is waiting. Release cycles happen quickly from the conception of the product through the final release. The philosophy is: “Do not chew more than you can swallow and if the piece is too big break it in smaller parts. And it you can not, then in parallel, pick at it one bit at a time.”

Develop small pieces, fast, integrate all the time and continuously test. The prototypes are key to this. Get QA in early, generating feedback along the way. Fix bugs early, do not wait until the end where there is a greater chance to have regression problems.

The same process can be applied to enhancements, except that if it takes longer than 2 weeks to develop an enhancement, then it should be considered a product release. Furthermore, there should be a team dedicated to bug fixes and patches where they crank a few bugs every day. The process remains the same, except in shorter increments of time and business requirements are replaced by bug reports.

The pipeline is the form that development takes once you have a high degree of parallelism in your technical organization. And if you can achieve a pipeline, then development will happen faster, with better quality, meeting your market on better footing and at the end, it should help you get ahead of your market. It does take a great deal of effort to set up the pipeline, but once in action, technology as a whole becomes easier to manage.

This is brilliant. Follow this link.

I often argue that what drives a company is a product strategy that is far seeing. In other words, a 10 to 20 years product roadmap that represents what a company wants to be, how it wants to be perceived in the market and what market position it wants to enjoy through execution of the product strategy. Most likely, however, in 10 or 20 years the company would be a different beast than what we can imagine today. I do not suggest that today we set in stone what products would be developed in 10 years, but I am indeed suggesting that a well documented plan needs to be in place. A well-documented plan drives overall company strategy and it allows, with a good process, change to meet the market in a very non-disruptive manner to the company.

For start-ups, on the other hand, they need to remain nimble, so a product plan should probably be 6 to 9 months worth; and as the company develops, the plan should expand. And eventually, if the company resolves the question of survivability, the product road map extends to encompass a wider more strategic scope.

So, here comes Apple with the iPhone. I think it is a great product. And on June 29th I got one. I was probably one of the first ones to get it in Los Angeles. But as great a product it is, it has problems. But those problems don’t really matter, what matters is that Apple took a pretty sizable risk. And if you think I am talking about going into the cellular telephony market, then you are mistaken.

There is a user-machine-interface paradigm change happening before our very eyes. New computing devices are appearing in the market that will forever change how users interact with computers and workstations. Microsoft Surface, a tabletop computing device is a good example. But it will not be available to consumers at home for a while. Multi-touch interfaces (go to this URL for a cool video of an application: http://www.ted.com/index.php/talks/view/id/65) is another good example of this paradigm change. Again, this technology will not make it to consumers for a while … but … wait a minute, the iPhone.

The iPhone is the first such device that has gotten massive consumer distribution; massive not because a lot of consumers bought it, but massive because it was targeted with a mass-market message.

For Apple, the iPhone was a well-calculated risk and there is nothing to loose from it. First of all, it was subsidized with an exclusive with AT&T. So, the R&D cost and probably a good chunk of the production cost is covered. Second, it is a great market research tool: deliver to consumers a product that they are used to, the iPod, but with a new interface and see how the market reacts. If the reaction is good, then the same interface can be deployed in more mainstream products, laptops or desktops, and we can say that the reaction has been very good for the most part. Third, it is clear that mobility in computing is core to their product road map. A mobile computing device, however incomplete, is also a step in the right direction and helps Apple morph its market position. It totally leapfrogged Microsoft.

With the iPhone, Apple is ushering and hastening the adoption on the new paradigm. There is no stopping the change, but whoever gets it right first, or at least almost right, will be in a great market position. And Apple is getting many things right to boot. But right does not mean success and the iPhone not making it huge is probably not too much of a concern for Apple.

A question that follows is whether Apple is the right company to bring this change. The answer, at least to me, is a resounding “YES”. Why?

If you look at Apple’s history of innovation you will get the answer to why I think it is indeed the right company. Apple brought most of the form and function changes that we enjoy today to market. Starting with the mouse, following with the Lisa, the Newton, workstation designs, Mac OSX (yes, I know, it is NeXTOS, I used to own one), the iPod and finally the iPhone; and in between all these, a bunch of incremental or leaping innovations and changes. Even within a product line, the iPod, the click wheel was significant change.

Does Apple care if the iPhone fails? Of course it does, what company would not? But probably what Apple cares the most is what the next 10 to 20 years will bring. The iPhone is a foray into those cares. A foray into deciding what kind of a company Apple wants to be in 10 to 20 years; into what products it will be offering and how it wants users to interact with its products. It is probably safe to say that the “hardware” keyboard is soon to be in our past. The thinner keyboards Apple is offering right now are just a calculated step into a software keyboard.

But what comes next?

I try to imagine how I would “write” in my Blog if there was no keyboard. To be honest, the only answer to that is dictation, and that has existed for a long time. So, I am sorry to disappoint you with, at least for the moment, with my lack of imagination. The change is coming, that is for sure.

Update:

The latest updates on Apple’s product line support some of my assumption in this post. They are using touch screen as a way to shift user-machine-interfaces paradigm.