stevedoria.net

Being commissioned for the development of a large software system caused me to reflect on the sources of past software production faults. Three elementary sources of problems were minimal or unacceptable project visibility, compressed schedule, and incomplete requirements analysis and design. These elementary problems resulted in secondary problems such as faulty software implementation, seemingly endless rework and refactoring, late delivery, lack of team coordination, demotivation or demoralization of development teams, and departure of team members.

Time and time management are the biggest factors in the success or failure of most software development projects. Underestimating the time needed for a software project guarantees late project completion. Attempting to complete a project that is not allocated sufficient time also results in corners being cut: design is shortchanged, testing is inadequate, and quality is hampered. Overestimation may discourage potential clients from proceeding with software development.

Estimating the duration of a project is an early opportunity for increasing project visibility. It encourages communication within the development team and allows the team to produce a project artifact, a somewhat detailed estimation on the resources needed for the project, early in the project lifecycle. Having the team develop resource estimates allows these estimates to be visible to all team members. These estimates serve as early deliverables that are given to clients to enhance their view of the project.

Delivery of the project artifact to the potential client gives the client more information to decide on continuing with the project. A prospect’s reception of a thoughtful time estimate, which is supported by consensus among the development team, gauges the prospect’s expectations. Prospects with reasonable expectations tended to be content when software systems were delivered on time, and they also tended to be moderately ecstatic when projects were completed before their respectively scheduled completion dates. Prospects with unrealistic expectations usually turned out to become problematic clients who demanded more attention and technical resources, causing a reduction in the quality of service provided to other clients. Clients with unrealistic expectations also tended to be insensitive to increased developer efforts to deliver high-quality software systems within reasonable time frames.

Requirements analysis is an essential tool for formulating an acceptable time estimate. Analyzing the desired system’s functionality requirements provides a concrete basis for estimating project duration. The complexity and time required for the development of software systems within a software class vary widely. The variation is based primarily on the differences of requirements. Listing a particular system’s required functionality and summing the estimated times to fulfill each of the requirements provides a more accurate estimate than one that is based on the system’s software class. Deriving a project duration estimate from system requirements analysis is an alternative to the common practice of making educated or whimsical guesses.

Like other artifacts of the software development lifecycle, the project duration estimate is affected by the completeness of the requirements analysis. Pockets of required time are hidden by abstract requirements. “Web pages are ranked by popularity on the ‘popular web page’ results page” is an example of an abstract requirement. This requirement is incomplete without coverage on the basis of popularity. The software system may use services like ping-o-matic or rely on a system component that spiders the Internet for information to determine web page popularity. The time required to fulfill the requirement, “popularity of pages is determined purely on recency information provided by ping-o-matic with the most recent page being the most popular,” is significantly different than that used to fulfill the requirement, “popularity of pages is determined by the number of people that bookmark a particular page on the social bookmarking service with the most popular web page having the most bookmarks.” Not eliminating abstract requirements causes project estimates based on the requirements to be inaccurate. It potentially causes the project and the delivered system to overrun its allocated resources.

Acquiring the requirements from the prospect, analyzing the requirements with the developers, and reviewing the requirements with the prospect after receiving feedback from the developers is a good start for developing a complex system. Requirements determine the time frame and form the foundation for system construction. Solid requirements provide a solid foundation for successfully building complex software on schedule. Incomplete requirements provide a shaky foundation that causes efforts to build potentially late software to topple.

Gmail’s “Why Use Gmail?” page suggests that “storing mail in folders with subfolders and nested sub-sub folders is not a productive way to spend your day.” Because search is Google’s primary strength, they encourage users of their free email service to perform a search query for an email of interest rather than actively organize their email and navigate through a folder hierarchy. In this way, Google is untraining many computer users who are so immersed in the file, folder, and cabinet paradigm of organization that they find other organization techniques incomprehensible.

Finding a particular email among the thousands that can be stored on Google’s email service can prove challenging, especially when more than 80% of all email falls under a small number of topics. Keyphrase queries fail to deliver results when items are not palpable to search engines. A typical email received by a network operator is presented here:

Meaningful queries to dig up this particular email seem to be “login,” “login info,” and “from:sales@gmail.com.” This being the typical kind of email received by a network operator, such queries would provide a large number of results.

The difficulty of tracking emails as encouraged by Google’s free email service has reached a level where certain technical teams have considered reverting to traditional email clients, such as Microsoft Outlook or Mozilla Thunderbird. After all, the teams are very familiar with the email folder schemes in these applications and organization has not been much of a problem before. These are proven email solutions after all.

One problem with traditional email clients comes to mind. Sometimes, email messages discuss multiple topics. These email messages, for example, may ask for a follow up on client XYZ while providing a task list for client ABC. In this scenario, the email is concerned with both ABC and XYZ. A system that uses the file and folder paradigm would require that the email message be filed into a folder for either ABC or XYZ. Using a more general folder such as “Tasks” or making copies of the email are also solutions. Taken to an extreme, the use of a general folder constitutes one folder, named “Email,” for all email. By reductio ad absurdum, the use of general folders is a less than optimal solution. Generating copies of an email is also suboptimal, because it requires more email to be filed by its very nature.

One feature of Google’s Gmail may help overcome organization difficulties that are faced when relying solely on search queries. Labels offer similar advantages to those provided by email folders of traditional email clients. They also overcome some disadvantages of email folders that were mentioned earlier.

Labels provide the grouping feature that is common with email folders. People who rely on a folder hierarchy for organization are still out of luck as Gmail has yet to implement a hierarchical system for email labels. They may be able to get around this limitation by setting a naming convention. Labels such as “CLIENT-abc” and “CLIENT-xyz” can be used. With labels, all emails that pertain to a particular subject can be viewed in a similar way that folders allow.

Emails in Gmail can be tagged with multiple labels. This allows an email that covers multiple and possibly unrelated topics to be placed in multiple groups. This feature is not directly supported by the file and folder paradigm. The time needed to label email messages may be comparable to that of filing email into folders, but the advantages of email labels should be sufficient defense against transitioning to traditional email clients on the basis of organization.

The Linux Programmer’s Manual states, “ldd prints the shared libraries required by each program or shared library specified on the command line.” This tool is useful when setting up a chrooted shell environment. It is also useful when creating custom boot CDs, which may include programs that dynamically link to shared libraries.

All things mechanical will fail. A lack of sound disaster recovery procedures should keep a knowledgeable IT administrator awake at night. Measures to prevent data loss are needed by many recovery scenarios and are a worthwhile vehicle to discuss the overall need to practice disaster recovery procedures.

Data backups are a key component of disaster recovery. Recovering from the failure of a complex system requires planning and training. IT administrators and operators should not be comfortable with simply deploying backup software. IT operators need to become comfortable only through continual practice of recovery procedures. They will also be better prepared to carry them out under the pressure of a real system failure. Recovering data will become less of an exceptional task and more routine.

Having IT operators routinely perform backups and recoveries will allow the operators to check the backup hardware. Checking the recorded data and verifying its recovery are necessary steps toward preparedness. Checking that data from older systems can be recovered onto newer systems may also be important. IT operators should be practicing different types of recoveries to verify that data can be recovered for different scenarios.

The routine practice of backup and recovery procedures also verifies the completeness and correctness of the recovery plans. It also provides the opportunity to measure their effectiveness and performance. Key measurements include the time needed for recovery as well as the amount of data lost between the most recent backup and the point of failure. Repeated validation of the procedures will provide opportunities for refinement. Special cases in the recovery process should be minimized for each system. Minimizing both time for recovery and loss of data can be worked on indefinitely.

Plan. Deploy. Practice. Repeat.

The desire of managers to be involved with technology is natural. They want to seem able to learn quickly and adapt. After all, failing to use new technologies puts a company at a disadvantage against its competitors. Effective use of technology can develop a small upstart into a best-of-breed market dominator. Management sees the potential power that technology has on the future of a business, and they are driven to master it before they are ousted through company politics by someone who is more able.

People, who are good at management, tend to ask more questions than suppositions when technical issues are not well understood. They are also able to provide complete, consistent, and detailed descriptions on what is technically desired. Although some entrepreneurs may have relied on gut instinct and got lucky, good managers develop instruments of analysis and employ measurements from these tools to form consistently good business decisions.

There are others in management who make assumptions. They raise vague technical issues, and they create an illusion of importance for these issues by keeping them vague and aligning them with general technical concerns. They worry about technology’s accessibility purely on intuition and arrogance. “We are like our target audience, and we cannot wield this technology, therefore this technology is inaccessible to our target audience,” is a logical deduction that is made commonly by weak managers.

Rather than make changes without measurements of performance, persuading people with weak management skills to gather information that supports their proposals is advisable. This helps weak managers to become more data-dependent. Analyzing performance data or deploying tools for such measurements is better than investing effort on changes based on personal intuition. Evaluating the outcome of any changes is difficult without proper tools in place.

Lead through suggestion, especially when not in a position to lead.