In October 2014, Victor John Yannacone, jr. and his colleague, independent software systems consultant, John Cona prepared and delivered a formal proposal to remedy the software problems of the OCME in real time and restore the OCME to the level of functional efficiency to which the people of New York City are entitled. The proposal was ignored and the Mayor compounded the problem by installing one of the individuals responsible for the debacle as Chief Medical Examiner. City newspapers continue to report more problems with lost and misidentified bodies. For that reason, the full proposal is posted here for consideration as the conclusion of this case study in data mismanagement and management ineptitude.
PROPOSAL
Introduction and Executive Summary
OCME IT Software Development has failed. Examples of that failure include:
- spending federal grant monies as well as NYC budgeted funds without quality delivery of software as bid and proposed
- very public failures of software systems and surrounding operational processes, resulting in cases of missing bodies, dysfunctional disaster modules, and underperforming case management system, gaps in chain of custody for body parts and evidence, and negative news coverage.
- lack of a feasible executive and program management structure supporting development and maintenance of software systems.
Proposed is a discovery, re-architecture, refactor, and software development initiative, to be directed and implemented by an expert team with significant advisory experience that understands OCME and software development, project management, software delivery and deployment who will complete the system and fulfill the vision of the “CMS” case management system and its supporting infrastructure—the System that the offshore contractor was paid to deliver in a fully operational state meeting industry standard levels of quality by this time; but which it has not.
This Proposal includes new strategies for:
- Sourcing personnel through cost efficient use of NYC-local resources, and American veterans).
- Contracting agreements for systems development and support.
- Pursuing new opportunities through value-added features built on existing Case Management operations.
- Components and structure of the proposed initiative
- Initial investigation and discovery
- Resolving critical security issues
- CMS operational and workflow stabilization
- Functional platform development for “Disaster Modules”
- Contract review, assessment, and remediation
- Operating and oversight models; establish a project management office (PMO)
- Foundation and strategy for new opportunities
A Cost Model and Executive Delivery Team biographies will be submitted on expression of interest.
Investigation and Discovery
The immediate first step in the development of the Data and Information Architecture for the OCME is to create a functional organizational chart of the OCME itself; then create an integrated function and process chart of OCME operations as they presently exist; as they have been mandated by a federal and state law, New York City regulations, and the identified exigencies of handling, identifying, and processing human remains in the city of New York; and how the professional staff at the OCME believe the system should operate over a reasonable foreseeable time span such as the next five years.
A true functional organization chart of the OCME is not publicly available, and does not appear to exist. An integrated functioning process chart of OCME operations has never existed. The need for both of these “charts” was overlooked by the offshore contractor and those at the OCME responsible for oversight and management of the offshore effort.
Security
More than thirty individuals have “Global Admin” rights which allow access to any and all “screens” and data. Most of these individuals are not employees of OCME and most are foreign nationals who work overseas.
There is a lack of formal processes and controls around assignment of access levels.
The “Audit Trail” functions of the CMS system are only partially implemented. Individuals can access cases with no record of them having done so.
Lack of a comprehensive audit trail coupled with liberal unrestricted access means that sensitive cases have distressingly open access even to non-employees, many of whom are offshore.
The team implementing the proposed initiative will:
- Oversee the discovery and repair of security breaches throughout the existing OCME IT system.
- Expand user security (LDAP, AD, Windows, and CMS application-based).
- Extend and repair audit trail subsystem.
- Analyze other CMS security lapses in basic processing areas such as the assignment and management of user roles.
- Analyze and correct access to the raw database tables to prevent, for example, editing of “production” (live, “golden-copy”, system-of-record) data when the intent and belief was that “test” or “staging” data was being manipulated. There are no controls of this process in place within OCME at this time.
CMS operations and workflow stabilization
Stabilizing operations and workflow requires initial analysis in a number of areas and from that analysis specific follow-on projects must be initiated to stabilize the core operations and workflow of OCME case management centered on CMS.
Data Architecture
Today, CMS systems reads do not produce reliable data. The lack of documentation from the offshore contractor means that every request for information becomes an exercise in doubt for staffers concerned with data integrity. There are very few who understand the database structure or the reporting code itself. There was an unfinished effort to test and secure data concurrency issues in January of 2014. The team making this proposal will complete the analysis and
- Document and refactor the Data and Information Architecture as necessary and stabilize data management processes to support improved reporting and audit functions.
- Thoroughly test auditing for performance issues, extended to cover all areas of CMS use, and roll out a new version to production with its use well understood by management and end-users.
- Refactor existing software to fix and/or document data concurrency design.
- Improve reference data management.
System Architecture
What currently exists is confusing, complex and unwieldy and has no useful documentation. CMS system performance is woeful and erratic. Web and middleware programming and architecture specialists unconnected in any way with the offshore contractor need to be deployed to develop and promulgate a plan for migration to a more stable technology foundation that follows industry accepted design patterns, standards and idioms.
After an expressed OCME need, a “technical review” of the quality of the software code delivered by the offshore contractor was delivered on February 6, 2014—by the offshore contractor.
The same occurred earlier with the “performance review” performed in October 2013, in which the offshore contractor found good performance of the application despite the ongoing degradation of the system to the point that it required a new release after numerous and widespread complaints from CMS users that screens implemented as web pages were taking several minutes to load, if they loaded at all.
“Performance Improvements” were undertaken in January and February, 2014 after the offshore contractor spent weeks in analysis and development of a new CMS release to improve performance and system latency. However, once deployed, the new release performed even more slowly and caused system-wide crashes throughout OCME.
The team implementing this proposal will complete the design and plan for implementation of adequate solutions.
- Restructure, reorder, and reorganize the CMS Business Rules Engine.
- Improve System performance and refactor code for to support scalability.
Workflow
Many chain of custody gaps are well known and were discussed in “CMS Advisory Board” meetings during 2013 and 2014, leading to investigations into these and other lapses in workflow and underserved reporting needs to support management of core operational processes. Solutions were proposed but never implemented. Currently there is no closed process for chain of custody within the CMS system resulting in errors and holes within the chain of custody.
The team implementing this proposal will complete the design and plan for implementation of adequate solutions to
- Fill all chain of custody gaps. For example, inter-borough transfers still have body and body part control gaps, as does evidence tracking and releases of remains to outside parties.
- Correct the process of moving bodies from one morgue (i.e. from the facilities at one city borough) to another.
Throughout 2014, there have been well-documented discrepancies in other CMS system reports, such as those that deal with City Burials. All such inventory, workflow, and chain of custody reports will be addressed by the team implementing this proposal.
The existing reports do not save their data, which makes looking across all inventory results in a comprehensive way—the way required of Mortuary and Executive Management—difficult, time consuming and error prone. The team implementing this proposal will
- present a specific design for saving this data, its timing and format, integrating it with existing CMS tables. The resulting reports will alert OCME management to lost bodies.
- Enhance “reconciliation” reports which show the result of inventories of remains taken by mortuary staff. To date these have been little used and even less trusted by staff and mortuary management because of their poor record of accurate performance.
- Even the simple improvements to these reports which have already been documented, but not implemented at OCME would have avoided regrettable cases such as the lost body of Rebecca Alper.
Platform for completion and acceptance of “Disaster Modules”
The “Disaster Modules” are CMS system enhancements which would extend the system to track data during disasters. In a post-9/11 world, these disaster modules will provide better preparedness and automated workflow for triage, body identifications, missing persons management, and family assistance center functions. The Disaster Modules were supposed to deliver new “day-to-day” functionality as well, for example in adding new screens to capture crime scene investigation data, and utilizing handheld scanner hardware for chain of custody support.
The team implementing this proposal will develop a stable and scalable functional platform enabling completion and acceptance of “Disaster Modules,” as well as all other enhancements going forward. This effort includes restructuring the Disaster Modules to meet user needs and demands in order to increase usability and functionality. The specific components to be supported are:
- An Automated Work Assignment Engine for all casework scheduling throughout the entire OCME workflow process. Currently this is not the case.
- Triage, Check-in and Tracking functionality.
- Investigator Desktop modules, including automatic scheduling and MLI scene report functionality.
- Investigator Mobile Site.
- AM/PM matching in accordance with the original specifications and requirements from OCME.
- Clustering of Missing Person/Reported Missing data to support NYPD as was specified originally by OCME.
- Modules to support Family Assistance Center, Call Center, and Credentialing.
- Functionality for OCME Pathology as originally specified.
- Support for multiple browser access to the system.
- “ZTR” (field-level auditing) improvements as originally specified.
- Scanner application support contributing to the full chain of custody solution originally specified.
- Expanded UDIM (dental system) functionality.
- LIMS and PHL integration.
- Request Response Fulfillment Unit workflow software.
- Workflow enhancements to support ME Triage and ID process improvements.
- Database management systems supporting Organ Retention.
Contracts Review, Assessment and Remediation
In order to properly repair all the damage done by inadequate and improper administration of the offshore contracts, a full review of those contracts is required with collection and assembly of all the material necessary to permit the City of New York to recover a substantial portion of the money paid to the offshore contractor for work which was not completed or does not operate as the contract required. This effort requires analysis and documentation of deviations from industry-standard best practices and quality measures attributable to the offshore contractor.
Discovery, due diligence, funding and contract review sufficient to permit the City of New York to recover some of the money spent for elements set forth in the original Scope of Work (SOW) and the Disaster SOW which have not been delivered in a form suitable for use throughout the OCME or that have been undelivered; developed with maintenance money; repaired or removed and thereby paid for twice; and which do not function as specified.
Establish operating and oversight models, and Project Management Office
Currently there is no Quality Assurance (QA) function at OCME within software development. The team implementing this proposal intends to design and implement a new management structure for software development initiatives at OCME in accordance with industry best-practices in Software Development Lifecycle Project Management and which will:
- Implement a PMO (Project Management Office)
- Develop Quality Control Function and Quality Assurance Processes which meet industry-accepted/acceptable standards and criteria.
- Prepare a complete project plan reflecting accurate project resources, achievable goals, and subordinate tasks. No tenable project plan exists at this time.
- Establish a formal project management methodology and a proper ignore alignment and management structure for OCME IT services to support software development and maintenance. No such feasible project management methodology has been established to this time.
- Provide meaningful and effective real-time support for all of the software modules.
- Provide technical documentation for revised system components including fully documented code and version control procedures.
- Provide sufficient user training in accordance with industry norms. Assure an orderly process for knowledge transfer from consulting work to in-house resources.
- Apply standard and industry-proven best practices for SDLC management, beginning with creation and separation of roles for the common duties: Business Analysts to liaison with the business-side representatives that sit on the CMS Advisory Board, Technical Architects for technical specifications, Developers, Database Architects, Project Managers, and a QA function, &c.
Executive Oversight, User Groups, and Steering Committees
Extant (2013/2014) oversight and leadership of CMS directives came from the “CMS Advisory Board,” Members of this Board are managers and executives who have qualifications in their own areas of expertise, but none in software development. This group also does not have the time to commit, nor the skills to apply, to take on the roles of Business Analysts and especially QA analysts and testers, though this has been expected of them.
The team implementing this proposal will
- recommend, develop, and implement a management structure that builds on the in-department IT SDLC structure for Project Management and enables oversight by higher level managers and executives at OCME, providing an operating model for efficient executive collaboration which will gather meaningful input from OCME managers representative of their particular expertise (e.g. Pathology, Anthropology, Disaster Operations, Forensic Biology, &c.)
- Establish a communication model which provide “rules of engagement” for liaison among groups and for decision making and setting of priorities which will drive delivery of software and services for case management.
- Separate longer term and strategic aspects of oversight and project direction from implementation and tactical project management.
- Establish User Groups and Community to repair IT’s current reputation for software development inexperience and to reposition Software Development within a Best Practice-based SDLC.
Code Management (Version Control) and Security and Roles Management
Currently, offshore contractor employees maintain the CMS and ancillary system source code repository, which should be owned and secured by OCME and New York City. Only direct employees of the offshore contractor know how to perform a system “build”, and deploy CMS to a new production environment.
Employees of the offshore contractor regularly make changes to the system code with no approval, documentation or user involvement. Indeed, with the existing failure of “security” and lack of control over the source code, there is nothing preventing them from doing so at any time.
There is very little system or database documentation, and none of what does exist is up to date. What training material exists speaks only to antiquated versions of the system.
Proposed is the implementation of best-practice based policies and procedures for code promotion, management, version control and maintenance.
New Sourcing
There are no OCME IT employees trained on core CMS application code which ensures that OCME remains beholden to and dependent upon the offshore contractor forcing New York City taxpayers to spend hundreds of thousands, if not millions, of dollars more to maintain the already inadequate CMS during upcoming months and years.
However, in the current labor market there are a great many talented Software Development professionals available in the New York City area who have substantial experience in each of the roles described within this proposal. These local experts will also bring pride of ownership to their service building software for their great city—pride and loyalty that is definitely not found with the offshore contractor.
Accepting this proposal will provide New York City with the opportunity to hire former or returning military personnel for many of these tasks and bring their discipline, work ethic, loyalty, commitment to public service, and skills in documentation, testing, analysis, and project management, in addition to technology to the service of their fellow New Yorkers. The initiative proposed will end reliance on costly offshore consulting companies.
New Opportunities
Once the platform is stabilized, untapped value can be found when the CMS vision is expanded.
Once a production version of the system is online, many opportunities arise for using the unique data that is captured.
Currently OCME scientists and researchers are required to work within the unintuitive, undocumented CMS database and system which relies upon scans of paper documents. Improved and automated data collection within CMS which will enhance research efforts is an element of this proposal. The team implementing this proposal will develop and specify initiatives that provide business intelligence and analytics capabilities from the CMS database. These include:
- Research. The world-leading physicians and scientists within OCME and their colleagues from throughout the world will be able to access advanced data of assured integrity and security suitable for accelerating ongoing research including, for example, SUID/SIDS, Dental ID metrics, Brooklyn College Infant Death Surveillance Study projects, Missing Persons ID, &c.
- Data Warehouse. A read only version of CMS which would not only provide an optimized reporting and data analysis environment, but would ease the load on the existing “live” CME data-gathering system while allowing much more productive use by the many who use CMS solely to read data (legal, ID researches, anthropology, ME use of scene reports, etc.).
- Reporting. Existing reports can be augmented and enhanced though the Data Warehouse. This includes both ad-hoc and “canned” reporting functions.
- Advanced and Predictive Analytics. The CMS data is of tremendous use, but it requires very sophisticated tools for even low-level data access and has a steep learning curve toward understanding the database structures given the unwieldy and unorganized development practices used during its development. Advanced users have used statistical packages such as R/SPlus and Wolfram Mathematica to develop mathematical models that provide everything from case arrivals modeling to time-series methods (e.g. GARCH) to standard descriptive statistics (suicides per borough, system usage, case type attribute regressions, &c.). The team implementing this proposal will complete the design and plan for a Data Warehouse and Analytics Platform to encourage and facilitate this type of development.
The possibilities are endless when considering integration with other data sources at OCME, in New York City, through independent agencies, access to modern behavioral and “big data” enrichment, and beyond. This is especially true given the 530,000-plus cases within CMS and its predecessors. The initiatives of this Proposal will support this kind of scalability of functionality.
Grants and Research
The team implementing this proposal will establish a dedicated structure to manage and procure Grant Opportunities based on the advanced Data Architecture and Management tools which will be developed under this proposal.