HomeMy WebLinkAboutBehavioral Health - Software ImplementationBehavioral Health - Software implementation report #13/14-3 May 2014
Behavioral Health – Software implementation
To request this information in an alternate format, please call (541) 330-4674 or send email to David.Givans@Deschutes.org
Deschutes County,
Oregon
David Givans, CPA, CIA, CGMA
Deschutes County Internal Auditor
PO Box 6005
1300 NW Wall St, Suite 200
Bend, OR 97708-6005
(541) 330-4674
David.Givans@Deschutes.org
Audit committee:
Jennifer Welander, Chair - Public member
Chris Earnest - Public member
Gayle McConnell - Public member
Jean Pedelty - Public member
Michael Shadrach - Public member
Anthony DeBone, County Commissioner
Dan Despotopulos, Fair & Expo Director
Nancy Blankenship, County Clerk
Behavioral Health - Software implementation report #13/14-3 May 2014
{This page left blank}
Behavioral Health - Software implementation report #13/14-3 May 2014
TABLE OF
CONTENTS:
HIGHLIGHTS
1. INTRODUCTION
1.1. Background on Audit …………..………………………………………...…… 1
1.2. Objectives and Scope ………………….……………………………...…… 1-2
1.3. Methodology …………………………………….…………………...……...… 2
2. BACKGROUND ………………………………………………………………… 3-4
3. FINDINGS
3.1. Compliance implementation status …………………………………….…. 4-9
3.2. Software implementation satisfaction observations ..………………….. 9-11
3.3. Software implementation status ……………………………………...… 12-16
3.4. Addressing compliance requirements ……………………………….… 17-18
3.5. Software controls ………………………………………………………… 18-19
4. MANAGEMENT RESPONSE – Deschutes County Health Services ...... 20-26
5. APPENDICES
5.1. Selected prior internal audit findings and recommendations ...….. 27-28
5.2. EHR Software implementation survey analyses ………………….. 29-38
5.3. EHR software implementation survey template ...………………… 39-42
5.4. Information on addressing compliance requirements ……………. 43-45
Behavioral Health - Software implementation report #13/14-3 May 2014
HIGHLIGHTS
Why this audit was
performed:
Initial discussions with
management indicated that
there was a lot of
dissatisfaction with the new
software.
What is recommended
Recommendations included:
assuring sufficient controls
to assure compliance
requirements are met
instituting sufficient
controls over services to
assure they are being
captured and billed.
enforcing a scheduling
solution,
utilizing software data to
improve billing and quality
management duties,
developing a coordinated
quality review process for
billing and quality
management.
assessing identification of
clients, exporting of client
records, and establishment
and handling of audit logs
Behavioral Health – Software implementation
Behavioral Health has been working since 2010 to bring about a conversion from their legacy record
system to an electronic health records system.
What was found
The current software does not have sufficient internal controls established to assure compliance with
regulatory requirements of serving the Oregon Health Plan participants under Medicaid. These
deficiencies could be significant. Analyses of identified compliance areas within assessments, treatment
plans and service notes highlighted some of the items that could be reviewed with the system data.
Compliance areas included timeliness of signatures; effective dates; documented diagnoses; and
expected amount, duration, and scope of each planned service.
A survey was performed to assess the satisfaction of the users of the system. Over the range of users,
service areas and software topics, the survey consistently indicated a very high level of user
dissatisfaction with the new software system. It was very difficult to find any satisfied ratings. Users were
dissatisfied with training provided on the new software.
“I have used a number of EHR systems in my career and this is by far the worst.
The amount of loss productivity has been substantial and we have spent a ton of resources
trying to make this broken system work. Our clients, staff and taxpayers deserve better.”
From comments in Appendix 5.3
Even one year after coming onto the system, there are a number of implementation areas that are being
worked on. Decision support (reporting) has not been fully implemented.
Staff productivity questions, raised in prior internal audits, appear to be still a concern. OHP billed
services provided as a percentage of funding received have been relatively low. In addition, staff billed
time per week is low. Staff are being plagued by software crashes that are disrupting the documentation
of services. It is understood that staff are sometimes creating non-billable time records for non-compliant
work.
The Quality Management Team is auditing through individual examination of electronic files in a process
similar to how they had performed audits of paper files. There is an opportunity to embrace new auditing
methods.
Software controls were reviewed and a couple of areas identified for further review. Those included what
audit logs were established and how they were handled and reviewed. Additionally, it was not clear there
were controls over client identification and exporting of clinical records.
Deschutes County Internal Audit
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 1 of 45
1.
Introduction
1.1 BACKGROUND ON AUDIT
Audit Authority:
The Deschutes County Audit Committee authorized the review of the Behavioral Health Software
implementation in the Internal Audit Program Work Plan for FY 11/13. Internal audit initially commenced work
in November 2012. Work was postponed until October 2013 when Behavioral Health transitioned to the new
software.
Current external audit of Behavioral Health
Deschutes County Behavioral Health is currently responding to an audit by DHS’s Office of Payment Accuracy
& Recovery (OPAR). OPAR is responsible for monitoring compliance with federal and state regulations for the
Oregon Health Authority and DHS. Deschutes County is one of a number of other Oregon counties and
providers that have been identified for review by OPAR as part of a standard review cycle. The audit was to
ascertain if patient records adequately documented the services billed and ensure practices in compliance with
Oregon rules. The period under review (within 2008) was prior to the County’s implementation of its electronic
health records software system. At the time of the internal audit, this audit had not been finalized.
The nature of the proposed findings relate to
invalid assessments (components missing such as signature)
invalid treatment plans (components missing such as listing services to be provided)
invalid or missing service notes
As these kinds of external compliance audits have increased within the state and nation, it will be critical to see
how these software systems have been setup to support compliant documentation.
1.2 OBJECTIVES and SCOPE
Objectives:
The audit objectives include:
1) Assess the current status of implementation in regards to deliverables established in RFP and contract.
a) Identify the status of the implementation.
b) Identify the areas documented as currently unresolved.
c) Identify survey opportunities to gather information on staff satisfaction with the new system.
d) Identify any unmet needs.
2) Assess the extent to which the EHR software system is addressing the following:
a) Compliance oriented treatment workflow (Assessment, Treatment plan, progress note) tied to gether
as expected?
DESCHUTES COUNTY
INTERNAL AUDIT
REPORT
DESCHUTES COUNTY
INTERNAL AUDIT
REPORT
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 2 of 45
b) Billing and encountering processes operating as expected?
c) Selected IT requirements working as expected? These should include general IT controls, audit log,
monitoring and reporting.
3) Be aware of any issues with compliance with fede ral and state regulations and requirements, as may be
applicable.
Scope:
The Software was initially implemented in October 2012. Internal audit work commenced in October 2013 and
went through February 2014. Software data obtained for analyses was from August 2013 through November
2013. For that period, there were around fourteen thousand records with indicated billings of $1.9 million.
Records analyzed were limited to billed services for certain cost centers. Findings developed from the
software system are limited due to sufficiency of data and time to program /develop the complex relationships
between the data.
1.3 METHODOLOGY
Audit procedures included:
Review documentation of RFP, contract and other implementation documents.
Interviews and meetings with staff and management regarding the software implementation
Development, issuance and analyses of survey on software implementation
Data acquisition and analyses of data within electronic software system. Time available was limited for
assessing complex relationships and was used to demonstrate how the data could be used and what it
might tell.
o These criteria are similar to ones used by the Quality Management Team in assessing services.
Areas identified to review included:
Assessments are present and valid for the dates of service. The assessment provides
appropriate diagnosis information.
Treatment plan provides the associated service to be performed and includes the frequency,
duration and amount of that service. The service provided is valid for the dates of service.
Service notes are signed.
We conducted this performance audit in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to
provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the
evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. (2011 Revision of Government Auditing Standards, issued by the Comptroller General of the United States.)
DESCHUTES COUNTY
INTERNAL AUDIT
REPORT
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 3 of 45
2.
Background
Table 1:
DCHS reported
data integrity
rates
In December 2011, the Behavioral Health department contracted with Uni/Care Systems (now called
Cocentrix) for an electronic health records software system referred to as Profiler. The overall cost of software
and hardware was anticipated to be over $400 thousand. Annual software maintenance costs are in excess of
$50 thousand.
Compliance and regulatory environment
Deschutes County Behavioral Health (DCBH) must follow the Integrated Services and Supports Rules (ISSR)
in providing Medicaid services. The ISSR specifies the minimum requirements in documenting clinical service
delivery. DCBH has a Quality Management Team performing compliance activities (data integrity audits) that
review for identified attributes over the services performed. Their reports are used for quality improvement
activities as well as assessing performance of staff.
Behavioral Health’s Quality Management Team has provided recent compliance reports that indicate an
improvement in compliance efforts.
Report
month/Yr.
Agency
Data integrity rate (%)
April 2014 87%
December 2013 84%
October 2013 80%
July 2013 65%
There is some indication the integrity rates may not reflect the overall population of the services due to less
than random sampling techniques utilized. It is encouraging still to see movem ent in the rate developed over a
short time span.
Prior Internal audit recommendations
Internal Audit has performed a number of reports with management recommendations to Behavioral Health. A
number of the recommendations were identified by management as having been addressed through the
implementation of the electronic health records system. Some of these reports date back to 2005.
Internal Audit Reports with Behavioral Health Recommendations
Mental Health Department – Review of Business and Contracting Practices (#04/05-6)
Follow-up report on Mental Health Department – Review of Business and Contracting Practices
(#05/06-11)
Health Services – Client Access to Care (#10/11-3)
Follow-up of Health Services Client Access to Care (#12/13-5)
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 4 of 45
Diagram 1:
Behavioral
Health workflow
A listing of some of the identified recommendations from past audits that have been identified as being
addressed through the software is included in Appendix 5.1. The findings relate to areas such as
Controlling service activities (scheduling, productivity, performance standards)
Reporting (management oversight and identifying billing errors)
It is not clear the implementation of the software has addressed these internal audit recommendations even
though the Department in a recent follow-up indicated that it had.
Clinical records workflow
Typically, a client who has been screened for services receives a comprehensive mental health assessment,
an individualized treatment plan, and services.
A clinical record must be maintained. Many of the County’s clients are served under the Oregon Health Plan
(funded by Medicaid) through a state contract. The contract and state provide guidance on the compliance
requirements necessary. The Integrated Services and Supports Rules (ISSR) govern all Medicaid services
provided by Deschutes County Behavioral Health (DCBH). Concerning the documentation of clinical service
delivery, the ISSR specifies the minimum requirements to meet Medicaid standards.
3. Findings These findings require some confirmation and work from Behavioral Health personnel. Though intended to be
complete and accurate, any identified compliance issues may be due to the underlying computerized data
techniques employed.
Management has fundamental responsibility for implementing systems designed to achieve compliance with
applicable laws and regulations. The audit was neither designed nor intended to be a detailed study of every
relevant system, procedure or transaction. Accordingly, the opportunities for improvement presented in the
report may not be all-inclusive of areas where improvements may be needed and does not replace efforts
needed to design an effective system of internal control.
Control deficiencies are deficiencies in the design or operation of internal control that could fail to prevent, or
detect and correct fraud, noncompliance with provisions of laws, regulations, contracts or grant agreements, or
abuse having a material effect on the financial statements or the audit objective.
Assessment Treatment
Plan Services
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 5 of 45
Software implementation staff and DCHS Management were open and professional and had a positive attitude
towards making improvements in the software system.
3.1 Compliance implementation status
Software system currently lacks effective internal controls to assure compliance.
The current software does not have sufficient internal controls established to assure compliance with
regulatory requirements of serving the Oregon Health Plan participants under Medicaid. Sufficiency is in terms
of the design and operation of controls over compliance efforts do not appear to allow management or staff, in
the normal course of performing their assigned function, to prevent, or detect noncompliance with provisions of
laws, regulations, contracts or grant agreements on a timely basis (excerpt from GAO Government Auditing
Standards §6.21). Some of these deficiencies could be significant in that they may have a material effect on
the financial statements of the Department and/or County.
The process of clinical workflow in behavioral health workflow normally begins with an assessment of the
individual, development of a treatment plan, and one or more services documented through services notes.
There are various types of State compliance requirements for the assessment, treatment plan and service
notes. Compliance objectives range from appropriate signatures to appropriate clinical care. A ny non-
compliance can jeopardize the associated billings.
Observations impacting internal control include:
The RFP, which should set the expectations for the software meeting compliance requirements, was
silent on the matter.
The software system is not currently designed to ensure compliance (as discussed below by area).
The preference would be for compliance to be dealt with proactively in the software though only
allowing compliant behavior. In the absence of proactive controls, reactive controls may be
implemented to assure identification of compliance issues. The latter is less efficient and requires
significant level of follow-up and administration.
The software system’s approach to handling assessments provides some challenges for effective
internal control as the treatment plan is sometimes disconnected from the assessment. This allows
inconsistencies to occur with work that will make it more difficult to be compliant.
The software’s configuration and setup make it difficult for clinicians to be compliant.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 6 of 45
During a software implementation of this magnitude, key risks should be identified and addressed in a
proactive manner. Relevant guidance can be seen from the financial and information technology fields.
Financial guidance can be seen from COSO (Committee of Sponsoring Organizations) who is an
acknowledged provider of guidance on establishing internal controls. Information technology management
and governance is addressed by ISACA (Information Systems Audit and Control Association) in their Control
Objectives for Information and Related Technology (COBIT). Proactive controls or controls that are assure
compliance are much easier to administer and are the most effective and differ significantly from reactive
controls such as reporting and continuous monitoring that yield information that needs to be addressed by
staff. {More information on this guidance is included in Appendix 5.4}
With average billed services amounting to $149, the impact of having to repay or reverse non -compliant
services can have a significant impact. With around a thousand services currently being billed a week, any
non-compliant services can result in a significant financial impact to the department and County. Recent
invalidations of billed services identified internally by DCBH amount to four percent (4%). Compliance issues
identified with assessments and treatment plans can potentially impact/invalidate underlying services. A
number of compliance type issues are discussed further below. However, there are significantly more
compliance areas than could be tested effectively during this review.
Aspects of the implementation that have contributed to the issues include:
The request for proposal (RFP) for the software did not properly outline the need for compliance with
state and federal rules pertaining to behavioral health as well as the need for release of information.
It appears the numerous and significant problems in getting clinical work established in the system
made some of the compliance aspects of implementation less of a focus during the long
implementation. It is understood the software system was made less compliant during implementation
by weakening many of the business rules that may have been established so that work could be
accomplished as staff learned the system. No additional controls were established to compensate for
loosening those controls.
It is also not clear that adequate resources have been employed by the department to fulfill the
requirements for such a significant software transition. The implementation team had no prior
experience in these installations and indicates little ongoing support or supe rvision from DCHS
Management.
It is recommended for DCHS management put in place sufficient controls to assure compliance
requirements are met whether they be through software design or through additional review by staff.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 7 of 45
Graph 1:
Trend in service
notes falling
within valid
assessment
period
Some compliance issues noted with assessments could jeopardize service note
compliance.
Analyses of assessments indicated some compliance issues.
Signature: Twenty-three percent (23%) of the in use assessments were signed over 7 days. Seven days
is their department requirement. Services can be invalidated if they are provided during a time when a
valid assessment has not been established. The data obtained indicated some thirteen percent (13%) had
insufficient information to tell whether they were signed (i.e. the audit did not receive sufficient data to
determine final signature). Impact would depend on the timing and number of services performed. Recent
data integrity audits have also brought up lack of signatures as a cause for invalidations.
Assessment effective period: On average, ninety-one percent (91%) of the assessments were effective in
the period the service was provided. This translated to an average pf eighty-nine percent (89%) of the
services billed were within the effective dates of an assessment. As shown in Graph 1, the observed rate
has improved steadily over the period reviewed.
One potential weakness in the current software system is that certain assessments are not effectively linked to
treatment plans or service notes. Recently developed assessment forms seem to better link the assessment
and treatment plan.
Observations from the audit data would need to be verified by appropriate clinical/administrativ e staff to assure
the findings. For the above assessment criteria, there was potential non-compliance impact for eleven percent
(11%) of the service notes.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 8 of 45
Some compliance issues noted with treatment plans could jeopardize service note
compliance.
Analyses of treatment plans indicated some compliance issues.
Signature: Thirteen percent (13%) of the in use treatment plan services were signed over 10 days by the
authoring clinician. Ten days is the department requirement. Services can be invalidated if they are
provided during a time when a valid treatment plan has not been established. On average, treatment plan
services are signed within 4 days. Impact would depend on the timing and number of services performed.
Recent data integrity audits have also brought up lack of signatures as a cause for invalidations.
Five-axis diagnosis: Documentation of a five-axis diagnosis is required. The system documents this in the
treatment plan section. Ninety-eight percent (98%) of the treatment plan services supporting billed service
notes appear to have met this criterion.
Expected amount duration and scope of each planned service: Often referred to as the frequency, duration
and amount of services. This is documented as part of the treatment plan. Ninety-eight percent (98%) of
the treatment plan services supporting billed service notes appear to have met this criterion.
Other areas: On average, these areas were found to be compliant with the treatment plans:
o effective date ranges for treatment plan effective date ranges,
o effective date ranges for the underlying services, and
o presence of Axis 1 Diagnoses.
Observations from the audit data would need to be verified by appropriate clinical/administrative staff to assure
the findings. For the above treatment plan criteria, there was potential non-compliance of six percent (6%) of
the service notes
Some compliance issues noted with service notes.
Analyses of service notes indicated some compliance issues.
Seventeen percent (17%) of the billed service notes were signed over 7 days (the department’s policy
requirement). Some 1% of billed service notes appear to not have been signed. A late or non-existent
signature could jeopardize or invalidate the services performed. The system requires multiple signatures
for items to become signed.
All of the compliance factors are assessed per service note, which creates a risk of one item making the
service note non-compliant as well as the impact from an issue at the treatment plan or assessment level that
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 9 of 45
will result in an invalidation of service notes. These contribute to the need for addressing compliance in a
proactive manner.
Developing an external audit assessment tool for this report would require more time and resources than
available for this audit. It is more effective to have the system designed to reinforce compliant behavior. The
alternative is continuous monitoring controls that require much more administrative oversight and may be
difficult to reinforce.
In addition, DCHS management and staff will want to assure a high level of quality with the documentation and
services provided. For this, they may identify other risk areas that warrant separate and distinct control
mechanisms. Having the software assure compliance provides for more effective use of the staff performing
quality management tasks.
3.2 Software implementation satisfaction observations
Graph 2:
OVERALL
satisfaction
ratings by
program
Survey finds overall dissatisfaction with software system in virtually all categories.
A survey of behavioral health staff using the new software indicated a very high dissatisfaction rating with the
current software system. We had a 51% response rate for the department. See Appendix 5.3 for more in-
depth survey results.
As indicated above, overall satisfaction ratings showed consistently dissatisfied ratings of the software. The
average overall satisfaction rating was 1.7 out of 5. In fact, it is difficult to find any areas where the average
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 10 of 45
Graph 3:
Average
satisfaction
ratings by
category and
area
Graph 4:
Training ratings
by Type
rating approaches a “satisfied” rating. Participants of the survey were consistently dissatisfied in each of the
software areas. Service note functionality is the only area with a less dissatisfied rating (better).
Participants of the survey were consistently dissatisfied with the software training areas
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 11 of 45
Other Survey observations:
In addition, it was noted that:
Seventy-three percent (73%) indicated they always use the scheduler. This seems somewhat contrary to
discussions with various department staff.
Fifty-two percent (52%) of respondents indicated moderate to extensive experience with EHR systems
outside of DCBH.
A number of other areas of dissatisfaction identified.
During meetings with staff on the new software, there were a number of areas they indicated were concerns
and included:
Productivity has been degraded by virtue of the additional amount of time this system requires.
Current software does not allow sufficient alerts and flags (i.e. current medications) to assure that safety
concerns are addressed.
Training and training materials are inadequate. Effective post training support has not been implemented.
Some of the software reports prepared for clients and shared outside of County are not properly formatted.
Software is complex and has a clunky interface with lots of mouse-clicking required.
Decisions made during the RFP and implementation process have contributed to the problems. Thos e
include:
the initial desire by management to have a very customizable product,
management’s expectations during setup created significantly more work in setups and maintenance (i.e.
staff should be setup to provide all services),
too little clinician input on software testing and decisions being made, and
inadequate testing of system before it went live.
In recognition of the issues being experienced during the software implementation, management now is more
open to a simpler product and is willing to discuss more standardized (plug-n-play) features.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 12 of 45
3.3 Software implementation status
Table 2:
Current
software status
areas
Software implementation work continues.
The implementation of the software started in 2010 and went live in October 2012. The software is still being
implemented.
Status of active areas still being worked in 2014 include:
STATUS DESCRIPTION
PROGRAM EXCEPTION The developmental disabilities program, which was originally included in the
RFP, was allowed to opt-out of the software. Deschutes County Information
Technology Department (IT) is working to develop a separate system for this
program.
OPEN Decision support software is installed but not implemented. This would provide
many of the required management reports.
WORK IN PROCESS 1. The "release of information" capabilities were not included in RFP and have
not been fully addressed.
2. Lab interface has not been completed by vendor. DCBH is withholding $56k
for work still to be performed.
3. Productivity reports are still in development.
4. Some audit trails have been initiated, but no one is reviewing or looking to
see what they contain and how they can be used.
5. "Meaningful use standards" are still in stage one of implementation. Staff are
still working on stage one deliverables. Stage 2 has not been started. The
County has received about $128 thousand as a subsidy from the Federal
government for work on this aspect of the program.
Software disruptions are numerous and impact productivity
During the audit, a survey tool was developed to gather information on the nature and extent of the software
problems being experienced by staff. Most notably, the software was freezing up and/or crashing. With those
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 13 of 45
Graph 5:
Trend in time
lost to crashes
Graph 6:
Crash reports
by type and
time lost
disruptions, staff sometimes lost important clinical documentation, which, in some cases, required a significant
amount of time to recreate.
In total, there was about 81 hours identified to these disruptions. As indicated above, the reported time lost
with these crashes has decreased in January 2014. DCHS Management believes that staff are not always
taking the time to report these disruptions and, as such, think there is considerably more lost time to these
crashes.
More than half of the crashes were attributable to the crash groups designated as “Lightning bolt” and “Grey
screen/frozen”. The lightning bolt crashes often occurred in the assessment process and resulted in a greater
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 14 of 45
loss of time per crash, since the clinicians often had to redevelop clinical documentation for the assessment .
The cause(s) for the software disruptions have not been resolved by the software vendor or staff.
Graph 7:
Trend in
encountered
service levels
as a percentage
of capitation
received
Staff productivity is still a question.
Prior internal audit recommendations relating to productivity still have not been addressed. These
recommendations related to productivity, scheduling, control for service tickets, as well as staff performance
measures (See Appendix 5.1). Many of these were supposed to be addressed through the new software
system. The department is still working on performance metrics.
The department receives capitation payments to cover anticipated OHP services. Capitation amounts are
supposed to balance with expected service delivery requirements. Based on the most recent information
available from the department, the level of provided services represents around 56% of the capitation
resources being received (most recent six-month average). The trend has improved significantly over the last
six months.
(Source: Pacific Source reporting)
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 15 of 45
Graph 8:
Trend in
average
provider weekly
billed hours of
service (with
high and low
bars)
There currently is not a method to assure all services are being captured. In the absence of sufficient controls,
it is difficult to assure that all services performed have been entered and are compliant. This in turn makes
managing and supervising staff workloads difficult.
As indicated above, the average productivity of billed hours is low. The apparent low average productivity
observed raises the question as to whether the department is getting credit for all of the services being
performed. Only 10% of staff have average billed services greater than sixteen hours. Billed time did vary a bit
by type of clinician.
Productivity is further impacted through a large number of unbillable services coded to “notes to chart”. It is
understood through discussions with staff, that this was the result of not being able to meet some of the
compliance requirements to bill the service. Over the course of August to November 2013, 680 hours were
charged to this non-billable service code. This represents 6% of service minutes that were billed.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 16 of 45
Graph 9:
Trend in use of
“notes to chart”
service code
The new software scheduling system has met with resistance and the department continues to struggle to get
all clinical staff to use the scheduling system as intended. Reception staff continue to identify disconnects with
staff and their schedules. There have been some software issues with the scheduler where staff have
inadvertently deleted appointments that were intended to be kept. Providing sufficient controls and measures
for managing productivity and service will continue to be difficult in this environment. This may influence future
funding levels as capitation is adjusted to reflect service levels reported and/or if revenues are more closely
tied to services billed.
It is recommended for the department to institute sufficient controls over services to assure they are
being captured and billed. To the extent needed, the department should gain a better understanding of
the extent services being provided by provider.
It is recommended the department find and enforce a solution to its scheduling needs that better
addresses client and organization needs.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 17 of 45
3.4 Addressing compliance requirements
Quality Management Team can audit more effectively.
Currently, the Quality Management Team (Team) is auditing through individual examination of electronic
files in a process similar to how they had performed audits of paper files. Their selection methods are not
random and may fail to adequately assess the overall integrity of the clinical records. Only a few of the
current audit criteria used by the Team are not determinable by data in the software system. Many of the
criteria could be identified as part of a fiscal or compliance check. Many of these criteria could be assured
through software system controls. The Team has recently been extracting data to identify and question
unusual service minutes.
Since the electronic files can be investigated through data analyses methods, it is possible to implement
auditing or continuous monitoring approaches using the underlying computerized data. These tests can
be all-inclusive and provide greater coverage of the underlying client records. Many of the tests that can
be performed are fiscal in nature and can be used to assure data integrity and basic compliance.
A computerized approach can leverage the time and work of the Team in a more effective manner. The
use of the software and data analytic tools can be performed on black and white compliance areas, saving
the efforts of the Team to provide more judgmental assessments of clinical documentation.
Continuous monitoring and computerized audit techniques are gaining wider acceptance as systems
become electronic. However, this is still a substitute to having a software system that assures compliance.
As previously indicated, invalidation work has resulted in a backing-out of 4% even though their calculated
integrity rates would indicate a larger problem. Some of this is because clinical staff must tell billing staff
what to invalidate and it is difficult for the Team to follow-up on the outstanding and non-addressed
findings.
It is likely that the long implementation time and the la ck of implemented controls within the system have
contributed to the need for a more piecemeal approach to auditing. The department has also been
struggling with inconsistent data from the system, which may be misinterpreted.
It is recommended the Department, after considering the integrity of the software product and data,
available staffing and the progress of implementation; coordinate the compliance efforts between
billing and the Quality Management Team.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 18 of 45
It is recommended the Department work towards utilizing the data in the software system to
develop a coordinated quality review process for billing and quality management.
It is recommended the Quality Management Team focus their audit efforts on quality and
compliance efforts that require more skilled assessment of clinical documentation once they have
addressed the more basic compliance efforts within the system.
3.5 Software controls
Audit log controls insufficient.
Audit logs assist in assessing whether the system is being used appropriately. Audit Logs can record
data changes within the software application so you can always find out Who, What, When, Where and
How. This information can also be useful in diagnosing problems.
Audit logs can provide information on
user access to the software,
identification of who entered the data,
records on exporting,
type of access,
corrections and changes to data,
data import,
disabling of the audit log feature, and
outside user access
Implementation staff have established a few audit logs and have not been able to identify what is in them.
No procedures have been established to review and monitor what happens with the audit logs. It is not
clear the appropriate logs have been established to address the most significant risks in the software.
Currently, the following audit logs have been created:
who creates a new record
who changes or creates an appointment
who accesses reports
In the absence of proper use of the audit log, it is difficult to determine how problems occurred and or
proactively diagnoses inappropriate uses.
Implementation staff have been reluctant to turn on too many audit logs as they may degrade performance
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 19 of 45
of the software. They also indicate that there are insufficient resources in terms of manuals, training and
staffing to handle the audit logs. There currently is no report to view and interpret the audit logs being
created.
It is recommended for the Department to establish appropriate audit logs and the underlying
oversight and reporting to assure the software is working as intended.
It is recommended that appropriate policies and procedures be in place for handling of the audit
logs.
General software control observations.
In general, the software appears to provide many of the general software controls expected of software of
this type. Those include providing for
user ids, authorization and security,
user access levels by role,
audit logs,
testing environment for software changes,
limited outside user access,
training on privacy, fraud, abuse and data
integrity, and
written policies and agreements for EHR users.
There were a couple of areas noted as potential weaknesses which included:
Front desk staff do not routinely identify clients and document the form of identification received .
Print screen option available by all staff.
Export function not limited.
It was not clear that staff has assessed the general software control environment.
Since the software is still being implemented, it is anticipated the control environment will continue to
evolve.
It is recommended the department assess how they could address identification of clients and
appropriate limiting of printing and exporting of clinical records.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 20 of 45
4.
Management
Response –
Health Services
Department
(Behavioral
Health Division)
DeAnn Carr,
Deputy Director
and
Michael Ann
Benchoff,
Program
Support
Manager
Deschutes County Behavioral Health Software Implementation
Management Response – April 2014
I. GENERAL RESPONSE
Deschutes County Behavioral Health is in overall agreement with the findings in this report. Management supports
the conclusion that the current electronic health system used by the agency is insufficient in meeting primary
business needs. Of particular relevance are the challenges associated with documentation and billing compliance,
ease of use of the system for staff, system failures, low customization features, slow and challenging
implementation process, auditing inefficiencies and general product dissatisfaction across all agency groups.
In response to these challenges, DCBH has been investigating alternative EMR . The agency is particularly
interested in the Behavioral Health module available in the OCHIN-EPIC system current used (to great satisfaction
and success) by the Public Health division of the Health Services department. This investigation has been robust in
its consideration of compliance controls, customization of templates and workflows, applicability to practice, and
clinician-friendly interface.
Responses to the specific findings identified in the report are provided below.
II. DETAILED RESPONSE
FINDING #1: Software system currently lacks effective internal controls to assure compliance.
Response: DCBH agrees with this finding. The inability to customize internal controls within the current EMR
creates additional workload for billing, direct service, and performance staff in order to protect the validity of claims
submitted for encounters and billings. Primary requirements being monitored include:
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 21 of 45
4.
Management
response –
(continued)
Assessment diagnoses fields completed
Assessment signatures
ISSP primary element fields completed
ISSP signatures
Assessment and ISSP expiration date alerts
Service note required element fields completed
Service note narrative and signatures completed
Currently, the majority of claims issues discovered in the quarterly Data Integrity Audit are caused by assessments
and ISSPs missing required elements or signatures. Comparatively, a small percentage of service claims fail based
on their own merit, independent of the assessment and ISSP to which they are anchored.
Within the current setup of the EMR, clinicians may inadvertently submit a claim for billing without complete
documentation. A software report titled “Progress Note Exception” report, highlights notes that are non-compliant
for one or more of the following reasons:
Note not final signed
No note started
Note narrative not complete
Any of these errors on a claim submission render that claim invalid. In order to address this issue, Quality
Management and Billing automatically back out claims for any of those identified notes not addressed by a clinician
within the stipulated timeframe.
DCBH has attempted to work with Cocentrix to put internal controls within the system to prevent these issues but
has been told that this is not an option within their current system. DCBH is now evaluating an alternative EMR
(currently used by the Public Health division of the agency) to determine the current compliance controls and the
availability of customization of such controls both prior to and after purchase of the system.
Summary of Actions:
Established a process for informing clinicians of missing claims elements through the Progress Note
Exception Report
Established a protocol for backing out claims with missing elements after X days of service
Outreach to Cocentrix to request EMR improvement
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 22 of 45
4.
Management
response –
(continued)
Currently in process of evaluation of alternant EMR system
FINDING #2: Some compliance issues noted with assessments could jeopardize service note compliance.
Response: DCBH agrees with this finding. For a service note to be compliant with Medicaid billing rules, it must be
anchored to a valid assessment. Data Integrity Audits performed by the Quality Management team have revealed
invalid assessments as one primary cause for the agency’s claim error rate. An invalid assessment affects all
services provided and once detected, the claims submitted during the invalid timeframe are backed out.
As stated above, since DCBH has been unsuccessful in working with Cocentrix to implement these internal controls,
the agency is looking at alternative software systems. In the meantime, ongoing chart audits and staff tr ainings are
the primary sources of prevention and detection of invalid assessments.
Summary of Actions:
QM team conducts Data Integrity Audits to identify issues with assessments. Protocols have been
implemented to back out claims associated with invalid assessments
Outreach to Cocentrix to request EMR improvement
Currently in process of evaluation of alternant EMR system
FINDING#3: Some compliance issues noted with treatment plans could jeopardize service notes.
Response: DCBH agrees with this finding. Similar to the assessment requirements, for a service note to be
considered ‘valid’ it must also be tied to a valid ISSP. Automatic auditing of the ISSP would allow or prevent
services from moving forward based on the completion of the ISSP elements. This is currently not an option with
ProFiler.
DCBH has not been able to modify the current ISSP structure within ProFiler to control these outcomes and is
currently relying on the Data Integrity Audit process to catch those ISSPs included in the audit sampling.
Summary of Actions:
QM team conducts Data Integrity Audits to identify issues with Treatment Plans. Protocols have been
implemented to back out claims associated with invalid Treatment Plans
Outreach to Cocentrix to request EMR improvement
Currently in process of evaluation of alternant EMR system
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 23 of 45
4.
Management
response –
(continued)
FINDING #4: Some compliance issues noted with service notes.
Response: DCBH agrees with this finding. Responses provided above address the concerns related to non-
compliant service notes. Without proper internal controls, there is the risk of non-compliance. Service note issues
are being identified by Quality Management audits and the Progress Note Exception report (described in the
response to Finding #1). These methods have assisted a great deal in improving the agency data integrity rate from
65% in October 2013 to 87% in April 2014 but have simultaneously resulted in backing out of OHP Encounter Data.
The Quality Management team has provided extensive training and resources to clinical staff in understanding the
required elements of assessments, ISSPs, and service notes. This, in combination with the audits, has been
responsible for the dramatic increase in quality documentation.
Summary of Actions:
QM team conducts Data Integrity Audits to identify issues with service notes. Protocols have been
implemented to back out claims associated with invalid service notes
QM team providing trainings
Outreach to Cocentrix to request EMR improvement
Currently in process of evaluation of alternant EMR system
FINDING #5: Survey finds overall dissatisfaction with software system in virtually all categories.
Response: DCBH agrees with this finding. Although the agency has utilized internal resources to improve the user
experience of the current EMR (including but not limited to: refresher trainings, system “Super Users,” weekly
workgroup meetings, use of forms used outside of the system, customized reports and regular consultation with
Cocentrix), the majority of staff express frustration and dissatisfaction with the system for a variety of reasons.
Summary of Actions:
The EMR workgroup meets on a weekly basis to troubleshoot issues with ProFiler
Trainings to improve user competencies
Outreach to Cocentrix to request EMR improvement
Currently in process of evaluation of alternant EMR system
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 24 of 45
4.
Management
response –
(continued)
FINDING #6: A number of other areas of dissatisfaction identified.
Response: DCBH has a mixed response to the itemized topics in this finding.
Productivity challenges: DCBH agrees that the software interface is more complicated than previous paper
chart practices and perhaps more so than other software options. It is not the opinion of all DCBH
managers, however, that ProFiler is the cause of the decrease in productivity since its implementation in
October 2013. Service analysis reports indicate great variation in productivity by clinician and by team and
do not support the allegation that ProFiler is the primary barrier for service outputs at this juncture.
Training and training materials are inadequate: DCBH has provided multiple forums for training both pre and
post implementation. Numerous factors are involved in the proficiency challenges with the system beyond
training opportunities and resources. At times, ProFiler has become the ‘scapegoat’ for other systemic and
personnel related barriers and issues.
Inadequate reports: DCBH agrees that both the assessment and the ISSP reports distributed to external
agencies have been incomplete and unprofessional in appearance. To address this and other issues
related to the assessment, the EMR workgroup created a “hung form” that lives outside of the main system
to ensure integrity of the report in addition to ease of completion.
Complex interface: DCBH agrees that the software interface is cumbersome and not user-friendly.
Summary of Actions:
The EMR workgroup meets on a weekly basis to troubleshoot issues with ProFiler
Trainings to improve user competencies
Outreach to Cocentrix to request EMR improvement
Currently in process of evaluation of alternant EMR system
FINDING #7: Software implementation work continues.
Response: DCBH agrees with this finding. Implementation and customization of the system, as promised upon
contract, have been slow-going and difficult to move forward.
Summary of Actions:
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 25 of 45
4.
Management
response –
(continued)
The EMR workgroup meets on a weekly basis to troubleshoot issues with ProFiler
Outreach to Cocentrix to request EMR improvement
Currently in process of evaluation of alternant EMR system
FINDING #8: Software disruptions are numerous and impact productivity.
Response: DCBH agrees with this finding. Work time lost due to software crashes is the source of high frustration
and duplicated work efforts for clinicians, which does impact productivity.
Summary of Actions:
The EMR workgroup meets on a weekly basis to troubleshoot issues with ProFiler
EMR staff provide assistance to recover lost data and/or track reasons for data loss in order to improve the
system
Outreach to Cocentrix to request EMR improvement
Currently in process of evaluation of alternant EMR system
FINDING #9: Staff productivity is still a question.
Response: DCBH agrees with the supposition of this finding but does not find it directly relevant to the overarching
issue of ProFiler. Many clinicians perform at an expected rate while others are performing significantly below the
standard. Actions have been taken to improve OHP encounter data including, but not limited to: individualized
team plans, development of a caseload management tool, and service utilization reports.
Summary of Actions:
Development of a caseload management tool that sets specific billing targets by clinician and team. This tool
will replace the historic productivity percentage standard
Trainings to improve user competencies
FINDING #10: Quality Management Team can audit more effectively.
Response: DCBH agrees with this finding. Much of the QM auditing time is spent on individual inspection of charts
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 26 of 45
4.
Management
response –
(continued)
for all ISSR compliance requirements. The use of automatic compliance controls within the electronic health system
is ideal. Such a function would automatically prevent a clinician from moving forward in the documentation process
without completing required fields. With an automated compliance system, the QM team could perform more
targeted and quality-based audits, improving the overall quality of services and resources available to clinicians.
Summary of Actions:
Outreach to Cocentrix to request EMR improvement
Currently in process of evaluation of alternant EMR system
FINDING #11: Audit log controls insufficient.
Response: DCBH agrees with this finding. DCBH acknowledges the need for appropriate policies and procedures
as well as regular utilization of audit logs.
FINDING #12: General software control observations:
Response: DCBH agrees with this finding. The agency recently added the position of Operations Manager to
improve overall processes and efficiency of facility operations including front desk, reception and medical records
activities.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 27 of 45
5.
APPENDICES
Appendix 5.1 Selected prior internal audit findings and recommendations
Selected findings and recommendations and management responses that were to be addressed through the
electronic health records system (Profiler) include:
Audit# Rec# Recommendation
Status as
provided New Comments
Estimated
Date
04/05-6 2
FINDING: The Department’s computerized business system cannot track and follow-up on outstanding service tickets with clinicians.
Department procedures for handling client services do not include any methods to track the receipt of service tickets from client appointments.
It is recommended for the Department to
develop a process to track and obtain
service tickets for all clients provided with
services. Complete
Service tickets are no longer used with the conversion to an
Electronic Medical Record (EMR). Current policy requires
documentation to be completed within 7 days. N/A
04/05-6 3
Same as above
It is recommended for Department
management to establish performance
standards for the turning in of service tickets
and monitor for open tickets. Complete
Service tickets are no longer used with the conversion to an
EMR. Current policy requires documentation to be completed
within 7 days. N/A
04/05-6 18
FINDING: Most clinicians do not schedule their appointments in the computerized business system. The client appointment data for 2004
indicated about 1 in 4 appointments are kept on the computerized business system. When clinicians do not use the appointments system,
reception staff are required to call clinicians when there are questions on their schedules. There have also been problems notifying clients
when clinicians do not show for work and their schedule is not in the system. Clinicians do not allow front desk staff to mak e appointments for
the clinicians in the system, which results in clinicians being responsible for placing their appointments in the system, which they are not all
doing.
It is recommended that appropriate mental
health staff maintain their appointments on
the computerized business system.
Reception staff should have the ability to
add, move and delete appointments in the
system. Underway
Since the full implementation of ProFiler on 10/31/13, all MH
clinicians have been directed to keep their schedule up to
date within ProFiler. Though some clinicians have done well
adhering to this new practice, others continue to maintain
separate schedules, which inhibit front desk staff from
identifying appointment times and clinician availability. A
memo was recently sent out reminding clinicians of this
expectation. Front desk staff have begun a tracking system of
those schedules, which remain empty, and supervisors have
been asked to "spot check" their staff schedules in ProFiler
periodically. 10/1/2014
TABLE 3:
Selected
Internal Audit
Recom-
mendations
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 28 of 45
TABLE 3:
(continued)
Audit# Rec# Recommendation
Status as
provided New Comments
Estimated
Date
04/05-6 27
FINDING: The current internally developed computerized business system has a limited number of reports and business office staff do not
appear to use them on a regular basis to monitor operations. Many of the reports are not designed to provide monitoring of the system and its
functions. The business team does not appear to use the reports consistently. IT staff develops reports as requested. However, more reports
are still needed to evaluate the performance of the system.
It is recommended staff develop procedures
to provide oversight of the Department's
activities as maintained in the computerized
business system. Complete
We now have a staff person assigned to verify our internal
controls. N/A
05/06-
11 1
FINDING: “Productivity” as defined by direct interactions between clinical staff and clients and seemed somewhat lower than the Department
standard of 55% based on documentation of clinician services.
It is recommended that management
consider tracking all client encounters
regardless of funding. Complete
All client encounters must be documented in a billable service
note or a note to chart. This is a standard of practice and is a
requirement for all behavioral health providers, regardless of
client funding source. N/A
10/11-3 12
FINDING: Schedule use – A number of clinicians were noted as not having their system turned on which resulted in front desk having to call
them every time an appointment comes in. It is still not clear that all clinician’s appointments are entered into the system, whi ch makes it
problematic to follow-up on paperwork.
It should be emphasized when the new
system comes online that staff will be
required utilize the new system for
appointments and documentation. Complete
Electronic Health Record system went live 10/31/12. All
appointments and documentation are made in the new
system. N/A
10/11-3 14
FINDING: Fiscal billing staff track errors on entry that are not reviewed by anybody. The fiscal billing clerk maintains information on errors in
billing records received and sent back. Issues noted include; duplicates, not signed, payer issues, client not opened, or other correction
needed. Three percent (3%) of the inputs had some error indicated. In addition, certain metrics are gathered by front desk staff that can be
calculated by the system.
It is recommended that some of these
metrics be incorporated in the electronic
records system if they are deemed relevant
to monitoring operations. Complete
Levels of acuity at entry are being documented and tracked
within the new Electronic Health Records system. Reports on
acuity and access timelines will be run monthly and reviewed
in the Behavioral Health Quality Council quarterly.
Additionally, these reports will be used to create quarterly and
annual performance reporting to the Central Oregon Health
Board. N/A
Auditor’s note: Even though many of these recommendations were identified as completed through
implementation of the new software system, it was clear during this audit that these require additional work.
These recommendations will be marked as requiring follow-up in the next follow-up cycle for the department.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 29 of 45
Appendix 5.2 EHR Software implementation survey analyses
Demographics:
By Job description: Clinician
Business/Fiscal/Administrative
services
Clinical
Supervision
Prescriber
or RN Management Total
Number of
Respondents 57 12 9 4 1 83
% of Respondents 69% 14% 11% 5% 1%
By Program: Adult Children & Family
Business/Fiscal/
Administrative Other Total
Number of
Respondents 36 34 10 3 83
% of Respondents 43% 41% 12% 4%
Overall Satisfaction:
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
OVERALL
Program: Adult
Children &
Family Other
Average
rating
Job Description:
Clinician 2.0 1.6 2.3 1.8
Other 1.5 1.0 1.8 1.5
Average rating 1.8 1.5 1.9 1.7
TABLE 4:
Survey
demographics
by job
description
by program
TABLE 5:
OVERALL
satisfaction
ratings with the
new EHR
software by
program
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 30 of 45
TABLE 6:
Ease of Use
satisfaction
ratings by
program and
software area
TABLE 7:
Ease of Use
satisfaction
ratings by
program and
software area
(by job
description)
Ease of Use satisfaction
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Ease of Use
Program: Adult
Children &
Family Other
Average
rating
Scheduler 1.9 1.6 1.7 1.8
Assessments 1.7 1.3 1.8 1.5
Treatment Plans 1.9 1.5 2.0 1.7
Service Notes 2.3 2.2 2.2 2.2
Medication Management 1.6 1.4 2.7 1.6
Business Functions 1.7 1.3 1.9 1.5
As indicated above, satisfaction with ease of use of service notes received slightly better ratings than other areas.
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Program: Adult Children & Family Other Average rating
Scheduler 1.9 1.6 1.7 1.8
Clinician 2.1 1.7 1.7 1.8
Other 1.5 1.4 1.7 1.5
Assessments 1.7 1.3 1.8 1.5
Clinician 1.9 1.3 1.7 1.6
Other 1.1 1.0 2.0 1.2
Treatment Plans 1.9 1.5 2.0 1.7
Clinician 2.0 1.5 2.0 1.7
Other 1.6 1.3 2.0 1.6
Service Notes 2.3 2.2 2.2 2.2
Clinician 2.5 2.3 1.7 2.3
Other 1.7 1.5 3.0 1.8
Medication Management 1.6 1.4 2.7 1.6
Clinician 3.0 1.3 1.6
Other 1.0 1.7 2.7 1.6
Business Functions 1.7 1.3 1.9 1.5
Clinician 1.8 1.3 1.5
Other 1.4 1.2 1.9 1.5
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 31 of 45
Diagram 2:
Ease of Use
word cloud of
comments
Word cloud of “Ease of Use” comments:
Wordle™
Selected quotes on “Ease of Use” comments:
This s a slow, non-intuitive system that does not communicate well with other systems.
I find the overall profiler difficult to navigate and time consuming with lots of layers to remember!
Again, the system is cumbersome, often slow, inflexible in how it can be applied, makes retrieval of
information slow and difficult, and has a frustrating number of little "quirks" which make it clinically a barrier
rather than a benefit.
It does not create efficiency, it is not intuitive, and does not function consistently.
The system is labor intense, not user friendly for clinicians and creates so much administrative work that it
deters from clinical time which should be our priority.
Technical/Clinical Performance
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Technical/Clinical Performance
Program: Adult
Children &
Family Other
Average
rating
Scheduler 2.1 1.7 1.7 1.9
Assessments 1.8 1.3 1.8 1.5
Treatment Plans 2.1 1.6 1.8 1.8
Service Notes 2.4 2.2 2.8 2.3
Medication Management 2.0 1.4 3.0 1.8
Business Functions 1.7 1.4 1.9 1.6
As indicated above, satisfaction with technical/clinical performance received slightly better ratings from business
and fiscal staff (Other) than other areas.
TABLE 8:
Technical/
Clinical
Performance
ratings by Area
by program
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 32 of 45
TABLE 9:
Technical/
Clinical
Performance
ratings by Area
by program (by
job description)
Diagram 3:
Technical/
Clinical word
cloud of
comments
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Program: Adult
Children &
Family Other
Average
rating
Scheduler 2.1 1.7 1.7 1.9
Clinician 2.3 1.8 1.7 2.0
Other 1.6 1.2 1.7 1.5
Assessments 1.8 1.3 1.8 1.5
Clinician 1.9 1.3 1.7 1.6
Other 1.4 1.0 2.0 1.4
Treatment Plans 2.1 1.6 1.8 1.8
Clinician 2.2 1.6 1.7 1.9
Other 1.8 1.0 2.0 1.6
Service Notes 2.4 2.2 2.8 2.3
Clinician 2.6 2.3 2.3 2.4
Other 1.8 1.3 3.5 1.9
Medication
Management 2.0 1.4 3.0 1.8
Clinician 3.5 1.5
2.0
Other 1.4 1.3 3.0 1.7
Business
Functions 1.7 1.4 1.9 1.6
Clinician 2.0 1.3
1.6
Other 1.3 1.4 1.9 1.6
Word cloud of “Technical/Clinical Performance” comments:
Wordle™
Selected quotes on “Technical and clinical performance” comments:
I used to be able to complete a full assessment in 1.5 hours and now it takes me close to 4 from start to
finish.
The system is very complex and not end-user friendly. Some of the things that should be easier with an
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 33 of 45
TABLE 10:
Documentation
time ratings by
Area by
program
TABLE 11:
Documentation
time ratings by
Area by
program (by job
description)
EHR system seem harder.
I feel like we took a huge step backwards.
…you are required to click multiple boxes to accomplish mundane tasks…
Documentation time ratings
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Documentation Time
Program: Adult
Children &
Family Other
Average
ratings
Scheduler 1.8 1.6 1.7 1.7
Assessments 1.6 1.1 1.6 1.4
Treatment Plans 1.8 1.4 1.6 1.6
Service Notes 2.2 2.0 2.4 2.1
Medication Management 1.8 1.4 2.5 1.7
Business Functions 1.6 1.3 2.1 1.6
As indicated above, documentation time satisfaction received consistent low ratings.
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Program: Adult
Children &
Family Other
Average
ratings
Scheduler 1.8 1.6 1.7 1.7
Clinician 1.8 1.7 1.3 1.7
Other 1.6 1.2 2.0 1.6
As sessments 1.6 1.1 1.6 1.4
Clinician 1.7 1.1 1.3 1.4
Other 1.3 1.0 2.0 1.3
Treatment Plans 1.8 1.4 1.6 1.6
Clinician 1.9 1.4 1.3 1.6
Other 1.5 1.0 2.0 1.5
Service Notes 2.2 2.0 2.4 2.1
Clinician 2.3 2.1 1.7 2.2
Other 1.8 1.0 3.5 1.8
Medication Management 1.8 1.4 2.5 1.7
Clinician 3.0 1.4
1.8
Other 1.3 1.3 2.5 1.6
Business Functions 1.6 1.3 2.1 1.6
Clinician 1.7 1.4
1.5
Other 1.2 1.2 2.1 1.6
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 34 of 45
Diagram 4:
Documentation
time word cloud
of comments
TABLE 12:
Quality of Care
ratings by Area
by program
Word cloud of “Documentation time” comments:
Wordle™
Selected quotes on “Documentation Time” comments:
Complex, bulky, cumbersome, inefficient computer program. Does not integrate well with workflow
resulting in decreased encounters and productivity.
Everything takes so much longer with this program. There is so much clicking, and so much going in and
out of each task in order to be able to do the next part.
The software does not help me do my job, I help it do its job.
Instead of saving us time, this EHR system is costing us time in most aspects.
Quality of Care ratings
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Quality of Care
Program: Adult
Children &
Family Other
Average
ratings
Scheduler 1.9 1.7 1.7 1.8
Assessments 1.9 1.6 2.3 1.8
Treatment Plans 2.0 1.6 2.3 1.8
Service Notes 2.3 2.3 2.3 2.3
Medication Management 1.8 1.5 2.0 1.6
Business Functions 1.8 1.1 1.8 1.5
As indicated above, quality of care satisfaction received consistent low ratings.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 35 of 45
TABLE 13:
Quality of Care
ratings by Area
by program (by
job description)
Diagram 5:
Quality of Care
word cloud of
comments
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Program: Adult
Children &
Family Other Total
Scheduler 1.9 1.7 1.7 1.8
Clinician 2.1 1.8 2.0 1.9
Other 1.4 1.2 1.5 1.4
Assessments 1.9 1.6 2.3 1.8
Clinician 2.0 1.6 2.3 1.8
Other 1.6 1.0 2.0 1.5
Treatment Plans 2.0 1.6 2.3 1.8
Clinician 2.1 1.7 2.3 1.9
Other 1.7 1.0 2.0 1.5
Service Notes 2.3 2.3 2.3 2.3
Clinician 2.4 2.4 2.3 2.4
Other 1.8 1.0 2.0 1.6
Medication
Management 1.8 1.5 2.0 1.6
Clinician 2.5 1.6
1.9
Other 1.0 1.0 2.0 1.1
Business Functions 1.8 1.1 1.8 1.5
Clinician 2.1 1.2
1.6
Other 1.3 1.0 1.8 1.4
Word cloud of “Quality of care” comments:
Wordle™
Selected quotes on “Quality of care” comments:
I am more focused on getting the system to work and that energy takes away from energy I need to be
formulated my clinical approach to supporting the clients and families t hat seek my services.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 36 of 45
TABLE 14:
Training ratings
by Area by
program
TABLE 15:
Training ratings
by Area by
program (by job
description)
This is not a system that set up to better serve clients. It's set up to get numbers for admin and it doesn't
even do that well.
The system has not helped in supporting our effort toward concurrent documentation and has hampered
quality due to the extra time it takes clinicians to navigate the system.
Training ratings
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Training
Program: Adult
Children &
Family Other
Average
rating
New Hire Training 2.1 2.1 3.0 2.3
Initial Implementation Training 1.8 1.9 2.6 1.9
On the Job Training 2.3 2.4 2.4 2.4
Formal Follow-up Training 2.1 2.2 2.6 2.2
How-to Materials 2.5 2.5 2.3 2.4
As indicated above, Training satisfaction received consistent low ratings.
On a scale of 1=Extremely Dissatisfied, 3=Satisfied, 5=Extremely Satisfied
Program: Adult
Children &
Family Other
Average
rating
New Hire Training 2.1 2.1 3.0 2.3
Clinician 2.6 2.0 3.0 2.4
Other 1.6 2.3 3.0 2.2
Initial Implementation
Training 1.8 1.9 2.6 1.9
Clinician 2.0 1.9 3.0 2.0
Other 1.3 2.3 2.4 1.9
On the Job Training 2.3 2.4 2.4 2.4
Clinician 2.5 2.4 3.3 2.5
Other 2.0 2.0 2.0 2.0
Formal Follow-up Training 2.1 2.2 2.6 2.2
Clinician 2.2 2.1 3.3 2.2
Other 1.7 2.4 2.2 2.0
How-to Materials 2.5 2.5 2.3 2.4
Clinician 2.5 2.4 2.3 2.5
Other 2.3 2.6 2.3 2.3
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 37 of 45
Diagram 6:
Training word
cloud of
comments
Diagram 7:
Scheduler word
cloud of
comments
Word cloud of “Training” comments:
Wordle™
Selected quotes on “Training” comments:
Initial training nearly useless. Learned by doing and asking while doing.
The initial implementation training was disastrous. It was rushed; trainers were just learning the system
themselves; many features did not work as intended; field testing was insufficient to warrant bringing the
system on line; and confusion reigned supreme.
We had to beg and plead for materials and training.
The follow up trainings have felt useless and not focused on the frustrations that we have with the system.
SCHEDULER
Word cloud of Use of scheduler comments:
Wordle™
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 38 of 45
Selected “Scheduling” comments:
• Difficult to schedule, time consuming, clients "fall off" the scheduler,…
• I always use outlook as my primary calendar as I don't completely trust the EHR program
• Other immediate demands cause me to put the scheduling issues on the back burner and ultimately
are often forgotten until just prior to the appointment time written down in the day planner.
Other selected quotes on “Insights into software” include:
Maybe next time, actual clinical staff should be a part of the effort to find a good system and we should be
able to use it prior to purchasing.
EHR should eventually be scrapped for a system that is simple to use, has included line staff in the design of
it, and yields the information supervisors need to track staff performance.
Some folks may be working fine within the new EHR, but I find it to be complicated, inconsistent and not very
clinician friendly.
I have used a number of EHR systems in my career and this is by far the worst. The amount of loss
productivity has been substantial and we have spent a ton of resources trying to ma ke this broken system
work. Our clients, staff and taxpayers deserve better.
It seems up to management to help the workflow and free -up clinicians as much as possible to allow for
billings and face-to-face services with clients.
It feels as though we have done as much as we can to make this system as good as it can be and it is still
failing us.
I've worked with several EHR programs in my career, and this is by far the most redundant and frustrating
one I have ever used!
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 39 of 45
Appendix 5.3 EHR Software implementation survey template
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 40 of 45
EHR software
implemen-
tation survey
(continued)
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 41 of 45
EHR software
implemen-
tation survey
(continued)
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 42 of 45
EHR software
implemen-
tation survey
(continued)
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 43 of 45
Appendix 5.4 Information on addressing compliance requirements
TABLE 16:
GAO identified
IT acquisition
critical success
factors
U.S. Government Accountability Office (GAO)
GAO also recently reported on the critical factors underlying successful IT acquisitions. Officials from
federal agencies identified seven investments that were deemed successfully acquired in that they best
achieved their respective cost, schedule, scope, and performanc e goals. Agency officials identified nine
common factors that were critical to the success of three or more of the seven investments.
Critical Success Factor (Top 5)
Number of
Investments
reporting
Program officials were actively engaged with stakeholders 7
Program staff had the necessary knowledge and skills 6
Senior department and agency executives supported the programs 6
End users and stakeholders were involved in the development of
requirements 5
End users participated in testing of system functionality prior to formal end
user acceptance testing 5
{Source: GAO-14-183T Information Technology – Leveraging Best Practices to Help Ensure Successful Major Acquisitions }
COBIT
COBIT is ISACA’s globally accepted framework, providing an end-to-end business view of the governance of
enterprise IT that reflects the central role of information and technology in creating value for enterprises. The
principles, practices, analytical tools and models found in COBIT embody thought leadership and guidance
from business, IT and governance experts around the world.
Enterprises and their executives use COBIT to:
Maintain quality information to support business decisions.
Achieve strategic goals and realize business benefits through the effective and innovative use of IT
Achieve operational excellence through reliable and efficient application of technology.
Maintain IT related risk at an acceptable level.
Optimize the cost of IT services and technology.
Support compliance with relevant laws, regulations, contractual agreements and policies .
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 44 of 45
COBIT helps enterprises to create optimal value from IT by maintaining a balance between realizing benefits
and optimizing risk levels and resource use. COBIT 5 provides a comprehensive framework that assists
enterprises to achieve their goals and deliver value through effective governance and management of
enterprise IT. COBIT enables information and related technology to be governed and managed in a holistic
manner for the whole enterprise, taking in the full end-to-end business and functional areas of responsibility,
considering the IT-related interests of internal and external stakeholders.
COBIT is supported by ISACA a non-profit organization that supports a broad range of IT governance
professionals.
{Source: Excerpts from COBIT 5 Executive summary}
COSO and the Integrated Framework for Internal Control
The Committee of Sponsoring Organizations of the Treadway Commission (COSO) has provided internal
control guidance through issuance of an Internal Control - Integrated Framework. COSO’s mission is to
provide thought leadership through the development of comprehensive frameworks and guidance on
enterprise risk management, internal control and fraud deterrence designed to improve organizational
performance and governance and to reduce the extent of fraud in organizations.
The Framework published in 1992 is recognized as the leading guidance for designing, implementing and
conducting internal control and assessing its effectiveness. The integrated framework for internal control
articulates five core principles of effective internal control.
A major deficiency in internal control represents an internal control deficiency or combination thereof that
severely reduces the likelihood that an entity can achieve its objectives.
More recently, the integrated framework has been incorporated into the Federal Green Book and OMB’s
Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards as a best
practice for internal control.
Behavioral Health - Software implementation report #13/14-3 May 2014
Page 45 of 45
Diagram 8:
Principles of
effective
internal control
(addressed in
integrated
framework for
internal control)
{Source: COSO Outreach Deck – May 2013 }
{End of Report}
Please take a survey on this report by clicking on the attached link:
https://www.surveymonkey.com/s/BH_Software_Implementation_1314-3