Losing bodies and wasting millions

A case study in software management failure:

Adapted from memos and reports delivered to the New York City Office of the Chief Medical Examiner by my colleague, independent software systems consultant, John Cona. Portions of the actual memoranda have been redacted because they are not relevant to the substance of this case study.

ME wasted $10M on faulty body-tracking software

The New York City Office of Chief Medical Examiner (NYC OCME) outsourcing failures

Millions of Homeland Security Department taxpayer grant dollars were sent out of the country via IT “offshoring”, but very little useful and effective software came back. Numerous mistakes in oversight and errors of judgment which resulted in the unforgivable misplacement and actual loss of bodies.

Many of the problems resulted from typical Software Development Management misconceptions; some were more general executive-level decision-making errors; and some were simply inexcusable failures in accountability and basic responsibility.

The root causes of events that damaged the image and reputation of the OCME; the breakdowns in process and leadership; and the ineffective practices and inefficiencies of OCME IT management are not unique to the New York City OCME. They are endemic in business, government, and non-government organizations (NGOs) throughout the world. Software Development must be managed in a robust way with competent and experienced people placed within correct job roles.

The OCME offshore contractor is not named and the country in which the contractor operates is not identified, not to protect the guilty, but because it is irrelevant to the substance of this open memorandum. The perils of “offshoring,” “outsourcing,” and “offshore outsourcing” will become obvious.

Another body lost”

As reported in New York daily newspapers, the body of a suicide who was brought to the Manhattan Morgue in September 2013 and whose trail was lost in mid-February 2014— is still “missing”. This has been a public embarrassment for OCME, resulting in firings, forced retirements, and suspensions of OCME personnel, many of whom were not responsible for the debacle which could have been avoided if even common sense management had prevailed within IT and Operations at the OCME.

At the center of many of the OCME problems was a Case Management System (“CMS”) , which was developed by an offshore third-party consulting firm, whose programmers and system architects were based in a foreign country overseas.

Genesis of the disaster

During the late part of 2013, there was another public case and cause for concern: the confusion between, and improper cremation of two different decedents with similar names . This led to a lot of talk at the CMS Advisory Board, a steering and management committee including numerous high level OCME Executives and the CTO (Chief Technology Officer) who was a former employee of the offshore contractor. The topic of many Committee meetings was how to avoid confusion of cases and errors in body “checkout” from OCME morgues to other facilities: to other boroughs, to Funeral Homes, to City Burial, to research medical schools, etc.
The CMS Advisory Committee identified lapses in workflow; chain of custody holes; software issues; and problems with reporting, process and workflow.

Unfortunately, as new topics and new “emergencies” rose to gain the focus of the Advisory team, there was no longer support for or follow-up on the results of investigative analysis or the suggestions already made.
As a result, the staff and management never noticed that the “missing” body had not been inventoried and remains essentially lost to this day!

Investigation

A preliminary forensic investigation was conducted by a number of OCME staffers and consultants whose domain-specific knowledge—their particular expertise and view of the OCME world—helped piece together a holistic vision of where data on this case might be found, how to access the audit trails, and any other information sources which might lead to a complete “forensic” data analysis.

With Hindsight…

Following the usual reaction of bureaucrats in trouble the world over, “When in trouble learned out; run in circles scream and shout,” it became obvious that there was a bit of a cover-up, an attempt at “white washing” the details of OCME operations concerning the handling of human remains. Consider:

• Executive management seemed more concerned with defending their offshore software development contractor than in finding the missing body!
• There was an immediate attempt to blame the mortuary staff but without any effort to examine the workflow in https://yannalaw.com/opinion/public-policy/losing-bodies/ mortuary and the practices and procedures imposed upon the mortuary staff by OCME management. Investigation showed that the mortuary staffers—low-wage workers in truly thankless roles—were given complex, convoluted, non-intuitive software of limited functional utility.
• While scapegoating Operations and Mortuary staff, IT culpability was completely overlooked.During a year of well-documented discrepancies in system reports such as those dealing with City Burials, the resources necessary to analyze problems and supply fixes were utilized elsewhere, a theme that can be found at the heart of most Software Development Management problems throughout the IT industry.

Though, With Foresight…

The CMS Reconciliation Reports which indicate the results of a physical body “inventory” taken within a given freezer at the morgue of a given city borough do not save and store the inventory data. Examining inventory results in any comprehensive fashion is difficult and unrealistic.

Millions Gone” The Wider Problem: The OCME CMS Advisory Board

OCME has paid—or, more accurately, American taxpayers have paid through Homeland Security federal grants—millions of dollars for a system whose few working parts should have only cost a small fraction of what was actually spent.

The primary “business sponsor” of these IT software initiatives within OCME made a number of mistakes, many of them business school “classics”:

• “Throwing good money after bad.” The OCME executives lacked the discipline and experience to know when to stop spending money on substandard work. Faced with constant promises of fixes and improvements from the time-billing offshore contractor led to lazy and short-term wishful thinking which maintained the status quo. The offshore vendor based in a foreign country was repeatedly admonished but never terminated. There was never any search for much less competition invited among possible local vendors and contractors qualified to perform similar work who might have had a better record of success and timely completion.
• “Incentivized to fail.” The offshore Contractor continued to fail because they were being paid more to fix their own initial poor software code than if they had written it correctly in the first place. What was delivered and rushed into production performed erratically and kept the local OCME IT staff, busy full time with support issues and patching software defects, all on the U.S. taxpayers’ dime.
• The “disaster modules” have been an unmitigated disaster. They were paid for by federal grants to support better preparedness and automated workflow for triage, body identifications, missing persons management, and family assistance center functions after 9/11. This money was also supposed to be used to add new day-to-day functionality including new information screens to capture crime scene investigation data and enable handheld scanner hardware to automate the chain of custody. They are still in development and do not do the job they were supposed to. The costs increase as the time to complete the work increases.
• “Performance Improvements” were attempted after numerous and widespread complaints from CMS users that screens implemented as web pages were taking several minutes to load, if they loaded at all. The offshore contractor spent weeks in analysis and in development of a new “release” to improve performance and system latency, but once deployed the new release performed even more slowly and caused system-wide crashes.There were other attempts to add functionality to the system where the additional code had to be removed at taxpayer expense upon user complaints of buggy or unusable performance.
• Ensuring that OCME remain beholden, exclusively, to the offshore Contractor. There are no OCME IT employees trained on the application code and OCME will be forced to spend hundreds of thousands, if not millions, more taxpayer dollars more to maintain the software in production in the upcoming months and years, if it ever works properly at all. Further, there is very little system or database documentation, and none of it is up to date.
• Worse: The offshore Contractor maintains the source code repository and only employees of the offshore Contractor know how to perform a system “build” and deploy the software to a new production environment.
• The offshore Contractor often assigns work to itself.
• Literally laughable mistakes. In a system screen that captures missing person data, the color of a missing person’s article of clothing is to be entered from a “drop down box” offering the user a choice of colors. The color “red” was not one of the choices.

And the Risks!

At OCME, as in many large government organizations that are not strictly controlled by military or intelligence agencies, there is a general disregard of basic security and accountability mechanisms. Some examples from OCME are:

• More than thirty individuals have “Global Admin” rights, which allow access to any and all screens and data. Most of these individuals are NOT employees of OCME and are foreign nationals who work overseas.
• The “Audit Trail” functions of the system are only partially implemented. Individuals can access cases with no record of them having done so.
• The combination of no comprehensive audit trail and liberal access means that the file contents of sensitive cases are distressingly open to non-employees, many of whom are offshore.
• There are also numerous security lapses in basic processing areas, such as the assignment and management of user roles.
• There are also many people with open access to raw database tables which permits inadvertent changes to “production” (live, “golden-copy”, system-of-record) data.
• The offshore Contractor regularly makes changes to the system code with no approval, documentation or user involvement. With the existing failure of “security” and lack of control over the source code, there is nothing preventing them from doing so at any time.

Typical Software Development Management Mistakes

There are numerous well-documented Software Development Management mistakes widespread throughout the IT industry. Many of these have been and still are being made by OCME.

Software Development—designing and writing “code”—is complex. It requires higher levels of logical and abstract thinking; yet also demands creativity. Communication skills are necessary in order to articulate and explain the many decisions made at all levels of design and implementation when following even the most detailed specifications. This makes Software Development decidedly more of an art than a science.
Managing this process is not simple and requires significant experience. It is certainly not “prescriptive” as is the case with many other typical IT Department functions, such as managing hardware, networks centers, operating system upgrades, server maintenance, etc.

Among the many well-documented and “classic” mistakes made by IT and Software Development managers that have occurred at OCME:

• Hiring low cost development talent. Software programming requires talents that cannot be learned readily a short period of time. An effective software programming effort requires bright individuals who also have experience. Cutting costs here is more expensive in the long run as mistakes accumulate.
• “Scope Creep” arises from lack of discipline and experience in managing deliverables and defining phased rollouts and iterative software releases. Preventing scope creep requires skilled Business Analysts who define requirements, and Project Managers who provide overall resource and task management within the Software Development Lifecycle (SDLC), working together with experienced Software Developers.
• Chronic underestimation of complexity within a software development project or initiative results in underestimating costs and timelines. Lack of discipline in succumbing to management or business pressures and lack of experience prevents rational allocation early in the planning phases for adequate time to deliver quality work. At OCME, just as in many large organizations, there is a culture of fear about delivering bad news and communicating shortcomings. Developers are hushed in the hope that the issues will fix themselves or somehow disappear over time.
• Ignoring user input. There is a widely held opinion among many executives that the users’ opinion on their own software needs is inconsequential. This is not just an obvious mistake; it ca be the deaqth knell of any software development project because programmers do not usually understand the business for which they are writing the software. It also builds animosity later when users react negatively to systems which are inappropriate to their needs and executive management attempts to “shove it down their throats”.
• Taking shortcuts in software development is always paid for, with interest, in later phases: during testing, Quality Assurance (QA), or once bugs are inevitably deployed to production. Both in code and in process, OCME Software Development management skirts formal methodology, ignores industry best practices, and shows an utter lack of discipline surrounding task tracking and change management.
• OCME, as many large enterprises, seemed to believe that they could replace formal, industry standard, tried-and-true approaches to process with their own version of management. For example at OCME, there are no Business Analysts or Project Managers, nor is there any Quality Assurance function. Code changes are mostly communicated by mouth or simplistic email to the offshore development staff, who write code and test it themselves, and then deploy to production. This breaks almost every empirical rule of successful SDLC methodology which has evolved over the last few decades.
• Worse. As a software crisis deepens, Managers and Executives many of whom have impressive qualifications in their own areas of expertise, but none in Software Development, step in to take over and solve the “problem.” These individuals, well intentioned as they might be, generally do not have the time to commit, nor the skills to apply, to take on the roles of Business Analysts, and especially, QA analysts and testers!Another classic mistake, so wonderfully described in standard SDLC texts such as the justly famous The Mythical Man Month by Fred Brooks, is to think “you can make a baby in one month with nine women”; that is, adding more people to software coding will speed up the process. Perhaps counter-intuitively, it will, in fact, slow the project down! Counter-intuitive notions like this abound in successful Software Development management, which is why it cannot be faked and requires substantial practical experience.

Additional OCME Management Mistakes

OCME IT management and CMS System leadership also made many mistakes that are not attributable to lack of specific SDLC expertise, but rather lack of common-sense-based management skills:

• All the eggs in one basket. The basket is the offshore Contractor, thousands of miles away!There is no division of duties and responsibilities. All decisions are made within the three hours weekly of Executive Advisory meetings and the rest is outsourced and offshored.There is no QA function. Coders should not test their own code, yet at OCME and in many companies they do.
• Vendor salespeople are invited to, and make presentations at, key planning and management meetings.
• Turnover of internal staff. When the CTO has not developed a comprehensive departmental plan, there can be no accomplishments except on a day-to-day, ad-hoc, emergency basis.

Modern Software Development usually requires a flexible infrastructure within which to function. Government agencies have numerous constraints. So do large publicly held corporations. Even within the constraints of money, time, and resources, there are strategies that will accomplish all the desired goals. The conventional corporate and government procedures to replace staff take too long. Budget limitations on software development are usually unrealistic at the outset and grow increasingly counterproductive as software development proceeds. “Red tape” and archaic formulations of organization policy, particularly the “Lowest Bidder” regulations limiting the appropriate use of consultants only further complicate an already complex situation.

Fixing “it” (IT’s Software Development Management and Delivery Function)

Management

Corporate and Organization Managers are, in general, very talented, accomplished, and respected within their particular domains. Where these teams of high-level professionals fail is in managing Software Development.
Many offshore Contractors are low-cost providers of brute-force development, lacking the sophistication to develop software of the complexity required of a web-based, distributed, real-time, multi-modal workflow, chain of custody, auditing, reporting, and case management system.

Simply put, the fix to many IT/Software Development Management problems is to hire an expert in Software Development and allocate sufficient resources with attendant accountability to get the job done. Someone who will think long-term, recognize enterprise priorities, and stick to plans.

Software development experts with the necessary experience will distribute the Software Development projects across many groups. That which is currently 100% within the purview of a single Contractor should be spread among internal Company employees, direct consultants, and more than one vendor, each applying their particular expertise. Standard and Industry-proven best practices for SDLC management should be adhered to, beginning with the creation and separation of roles for the common duties: Business Analysts to liaison with the business-side representatives; Technical Architects for technical specifications, Developers, Database Architects, Project Managers, and a QA function. This is a canonical minimum.

The talent is here at home not in some foreign country!

In the current market there are a great many talented Software Development professionals available in the United States that have substantial experience in each of the above roles. Local experts will also bring pride of ownership to their service—a pride and loyalty that is definitely not found in offshore groups. Of particular note today is the opportunity to hire former or returning-military personnel for their discipline, work ethic, loyalty, commitment to public service, and skills in documentation, testing, analysis, and project management, in addition to technology. A little creativity—along with the appropriate experience in IT Management—can solve problems in the face of any bureaucratic constraint.

To examine the authors’ proposal to repair the damage to OCME operations and public image, please click here.