I ask myself sometimes disturbing questions like "You know the software business. You know how systems are built. Are you now comfortable, sitting in this airplane, 10km high in the sky, flying on systems that some software geezer constructed?" Uuuhhhr, honestly I don't know. The best thing is not to think about it too much.

What I know is this: after designs are made (or even during) software engineers have to try things out. They are not always sure that certain concepts can be done at all, or wonder how it will act in reality. To stay in the language used in this course: they need feedback on the parts and principles they are supposed to implement. And these trials are not always thrown away… like they should.

Pilot, prototype and Hollywood

Brooks has a very clear opinion on this matter:

The management question, therefor, is not whether to build a pilot system and throw it away. You will do that. … Delivering that throwaway to customers buys time, but only at the cost of agony for the user, distraction for the builders while they do the redesign, and a bad reputation for the product that the best redesign will find hard to live down. (The Mythical Man-Month: Essays on Software Engineering, 20th Anniversary Edition)

You will create pilots for trying things out, but the pilots are focussed on this particular aspect that they are attempting to prove. They are not designed as a part of a whole, which makes them a nightmare to use in a complete system.

But before throwing them away, these exercises are crucial for your project. They provide at an early stage the first glimpses of the requirements in a product. If you take a trip to Hollywood you should visit the Universal Studios. There you can be guided through the studios. You will see all those little streets you can see in movies and TV series. Even standing before a building, you can appreciate the look and feel, you can sense the ambiance. If you turn the corner, you see that the street consists of only the front of the houses. Disappointed? No. The mock-ups serve their purpose.

A prototype is a mock-up of software, a Hollywood system. You can use it in an early stage to give users a sense of what the end will look like. Mostly it's just a window with buttons that don't do anything. But a user can mentally walk through the system. Don't forget to tell them it's a mock-up, to avoid later disappointment. Feedback time! And if they like it, get it signed!

Proof of concept and benchmarks

A couple of years ago I did a project where we implemented a system that consisted of multiple remote databases that had to be synchronized. Perhaps for you that is peanuts, but at that time we had no experience in this concept. Ok, it was done elsewhere, so it had to be possible. The supplier of the software had done a small exercise in this area with an administrative system they used internally for tracking the projects. It consisted of one central database and small databases on laptops of the engineers. This provided a perfect feedback to the customer that the concept could be done, a so called, proof of concept.

However, skepticism arose on the scalability of the concept. It might perform well with a few users and data, but how does this perform with heavy data traffic and hundreds of users? This matter was handled by building a pilot(!) that just synchronized databases with a lot of data, and a simulated workload of a user making updates and queries. The time it took for results came back for the simulated user, and the time it took to synchronize (and some other technical measurements) were used as a reference for the future system. It was considered a benchmark. Those measurements on the pilot were acceptable, so the system to build was regularly checked against the benchmark. It provided a feedback on the scalability and on how far the actual system was away from the acceptable measurements.

Plan it

The activities described on in this section, they will be performed. It is how software people must work, and it is how you can provide proper feedback to your stakeholders. Listen to Brooks: don't deliver the throwaway as a final system, you will lose in the end. It will happen, so plan it in advance, make it an official part of your project… Plan time to try things out, you will, anyhow!

Links of Interest
There Is No Box: Agile And Plan-Driven Project Management Do Not Exist