HomeMy WebLinkAboutVeterans Services-EffectivenessVeterans’ Services - Effectiveness report #12/13-1 February 2013
Veterans’ Services –
Effectiveness
To request this information in an alternate format, please call (541) 330-4674 or send email to David.Givans@Deschutes.org
Deschutes County,
Oregon
David Givans, CPA, CIA, CITP, CGMA
Deschutes County Internal Auditor
PO Box 6005
1300 NW Wall St, Suite 200
Bend, OR 97708-6005
(541) 330-4674
David.Givans@Deschutes.org
Audit committee:
Gayle McConnell, Chair - Public member
Chris Earnest - Public member
Jean Pedelty - Public member
Greg Quesnel - Public member
Michael Shadrach - Public member
Jennifer Welander - Public member
Anthony DeBone, County Commissioner
Dan Despotopulos, Fair & Expo Director
Scot Langton, County Assessor
Veterans’ Services - Effectiveness report #12/13-1 February 2013
{This page left blank}
Veterans’ Services - Effectiveness report #12/13-1 February 2013
TABLE OF
CONTENTS:
HIGHLIGHTS
1. INTRODUCTION
1.1. Background on Audit …………..…………………………………………. 1
1.2. Objectives and Scope ………………….………………………………… 1
1.3. Methodology …………………………………….………………………… 2
1.4. Background on Veterans Services …………………………………… 2-3
2. FINDINGS and OBSERVATIONS
2.1. Customer Survey Observations …...……………..………................. 4-7
2.2. Operational Findings ……………………………………….……...… 7-12
2.3. Fiscal and Budgeting Observations ……………………….……… 12-13
3. MANAGEMENT RESPONSE
Veterans’ Services Office …………………………………………...… 14-17
Veterans’ Services - Effectiveness report #12/13-1 February 2013
HIGHLIGHTS
What is recommended
Recommendations
include:
developing a policy to
address extent of
services provided to
non-county veterans;
developing plans to
reduce backlog for new
client appointments;
utilizing to a greater
extent the veterans’
software;
considering additional
performance measures
that are useful in
addressing the level and
quality of services being
performed;
considering the means
to more efficiently collect
performance data;
considering whether
remote locations are the
best use of limited
resources; and
evaluating addition of
additional .25 FTE to
assistant VSO position.
Veterans’ Services - Effectiveness
Why this audit was performed:
Veterans’ Services has seen an increase in their new client backlog in providing services to veterans.
New clients are experiencing 2-3 month delays in being seen. The internal audit was requested from
management to see what could be done to improve the backlog to see new clients.
The Deschutes County Veterans’ Services Office (the Office) provides a valuable service to veterans
(including spouses and dependents) needing to obtain federal benefits. The Office provides necessary
legal and representational services to advocate for requested claims.
What was found
A customer service survey performed indicated very high satisfaction with the services provided. A
modest decline in satisfaction with timeliness for getting an appointment was noted, particularly with new
clients.
The backlog of new clients (currently 60-90 days) is a concern for a department with limited resources.
The Office currently provides services to non-County veterans which contributes to the backlog. Nearly
12% of their clients are non-Deschutes County veterans. Personnel turnover has contributed to a
decline in new client appointments per day. New client appointments are scheduled primarily over two
days.
The Office has been utilizing a new software system since 2010 for preparing and managing their veteran
case load. They have not used it fully which limits the effectiveness and efficiency of the system and
does not capture the extent of their work or results.
Performance measures used by the Office are primarily output measures provided to the Oregon
Department of Veterans Affairs (ODVA). There are opportunities to improve the collection of information
and the measures developed.
Veterans’ Services operates remote locations in Redmond and LaPine that end up costing the Office
resources. The VSO indicates many veterans from the currently served remote locations come to Bend
anyway to be seen. They would like to consider using these monies to increase personnel levels for one
VSO position.
Deschutes County Internal Audit
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 1
1.
Introduction
1.1 BACKGROUND ON AUDIT
Audit Authority:
The Deschutes County Audit Committee authorized the audit of the Veterans’ Services in the revised Internal
Audit Program Work Plan for 11/13.
1.2 OBJECTIVES and SCOPE
Objectives:
The objectives were.
Survey customers to evaluate levels of customer service.
Identify efficiencies and effectiveness in the Office’s process for processing of clients.
o Identify the ways client services are provided and documented (use of software, appointment times,
messages, and emails). Review how duties are shared within organization.
o Look for relevant measures of performance in this area (primarily time).
o Look for opportunities to streamline and combine activities in this process.
Analyze information for clients being served and relevant measures, including trends and benchmarks
o Activities by location and impact to time for travel.
o Activities by staff person
Scope and timing:
The audit commenced in November 2012 and continued through December 2012. Data, when available,
was reviewed for this and prior periods. For the audit, we defined new clients a little more broadly, as those
clients that are new to the office and/or filing a new claim. Even with these distinctions being made, it is not
clear we have an accurate picture of all activity from the software systems. The activity we were able to
quantify should give a reasonable approximation of levels and tren ds. As noted in the findings below, data
utilized was somewhat limited to the extent of data entry that occurs into their software system. Data quality
will limit the extent and quality of the findings. Recommendations and observations have noted these
limitations.
The scope of the audit did not include all aspects of internal controls employed.
DESCHUTES COUNTY
INTERNAL AUDIT
REPORT
DESCHUTES COUNTY
INTERNAL AUDIT
REPORT
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 2
1.3 METHODOLOGY
Audit procedures included:
Observing and interviewing staff on operations,
Developing, performing and analyzing survey of customers,
Collecting and analyzing timesheet and calendar information,
Extracting and analyzing information collected within veterans’ services software system (VetraSpec)
Identifying a means to identify new client activities from other activities.
Reviewing written procedures and documents provided,
Reviewing and analyzing revenue and expenditure data, and
Obtaining and analyzing customer locations.
We conducted this performance audit in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to
provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit
objectives. (2011 Revision of Government Auditing Standards, issued by the Comptroller General of the United States.)
1.4 BACKGROUND ON VETERANS’ SERVICES
The Deschutes County Veterans’ Services Office (the Office) provides a valuable service to veterans
(including spouses and dependents) needing to obtain federal benefits. The Office provides necessary legal
and representational services to advocate for requested claims. Much of this is a legal process and
requires initial claim work, claim development, follow-up and routine updates. Many of the claims continue to
require work through the life of the veteran, veteran’s spouse, and/or veteran’s dependent.
The Office serves a population of veterans numbering over 15 thousand. Currently, the office is handling
claims activities for 3,300 veterans. The Office is responsible for over $2 million in recovery awards received
in FY 12 that benefitted directly the veterans they serve. There are additional monthly awards on top of
those. They conducted over 2,000 interviews in 2012.
DESCHUTES COUNTY
INTERNAL AUDIT
REPORT
DESCHUTES COUNTY
INTERNAL AUDIT
REPORT
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 3
The Veterans’ Service Officer (VSO) is the primary provider of services. VSO’s are highly trained employees
on the specific legal and process requirements from the Oregon Department of Veterans Affairs (ODVA).
Training and development of these officers takes a couple of years and a lot of on the job train ing. The
County currently has one VSO and one assistant VSO.
The audit noted that the Office operates with a substantial backlog of clients. A new client today may have to
wait 90 days or more for an appointment. Claims for benefits start and become effective with an application
for those benefits. The Office makes efforts to file temporary claims for clients awaiting appointments but it is
not clear that this occurs for all of them. New clients perhaps create the greatest workload. New clients, in
addition to an initial appointment, generate at least two follow-up appointments, phone calls, emails and the
necessary time to develop claims which include requesting and assembling relevant medical records and
service history.
The Office anticipates that their workload will only increase as the economy and ending military deployments
increase the number of veterans and the likelihood of them needing assistance.
The Office operates with at $267 thousand budget and receives some 23% of its funding from the state.
Most of their support is from the County’s General Fund.
2. Findings
and
Observations
The audit included procedures to assess customer satisfaction and assess how the Office could be more
effective. Audit findings result from incidents of non-compliance with stated procedures and/or departures
from prudent operation. The findings are, by nature, subjective. The audit disclosed certain policies,
procedures and practices that could be improved. The audit was neither designed nor intended to be a
detailed study of every relevant system, procedure or transaction. Accordingly, the opportunities for
improvement presented in the report may not be all-inclusive of areas where improvement may be needed
and does not replace efforts needed to design an effective system of internal control.
A significant deficiency is defined as an internal control deficiency that could adversely affect the entity’s
ability to initiate, record, process, and report financial data consistent with the assertions of management in
the financial statements. The findings noted were not considered significant deficiencies.
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 4
2.1 CUSTOMER SURVEY OBSERVATIONS
Chart I -
Summary of
satisfaction
responses by
respondent type.
Customer survey results were positive.
The customer survey was carried out over November 2012 with ninety-two respondents. Customer service
observations included:
Overall customer satisfaction over the survey was 4.9 (1=low and 5=high).
This was consistent across all categories of clients and regardless of how long it took for clients to get
their appointments. There were no respondents with scores less than a 4.
The average number of days to get an appointment was consistently higher for new clients than other
clients.
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 5
Chart II -
Days to
appointment by
client type
100% of the customers would recommend this Veterans’ Service office to their friends and family.
Other observations included:
Most of the respondents who indicated how they heard of the service had come in from referrals from
other veterans.
Over 80% of the respondents were veterans (as opposed to spouses of veterans and dependents of
veterans)
Analyses of customer survey comments indicate mostly favorable descriptions.
A word cloud of frequently used words by the survey respondents gives some indication of the sentiments
carried with customer comments in the survey. In general, they conveyed their gratitude for the help of the
staff.
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 6
Chart III -
Word cloud of
customer survey
comments
(Top 25
frequently used
words)
Diagram through www.Wordle.nettm
Chart IV –
Days to
appointment
average and
Timeliness
satisfaction
by type of client
visit
Some indication of decline in timeliness satisfaction with longer wait times for
appointments.
Customer satisfaction for timeliness, as anticipated, declines with the number of days to obtain an
appointment. In particular the new clients are subject to significant declines in satisfaction. However, those
declines seem to start after 30 days.
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 7
Chart V –
Timeliness
satisfaction trend
by days wait for
appointment
(size of circle =
number of
respondents)
As evident in the above charts, new clients seemed to have the greatest dissatisfaction in timeliness and they
experienced, on average, the longer times to appointment.
As would be somewhat expected, satisfaction levels declined with average days to employment. The
satisfaction decline was more significant and more prevalent with new clients. The audit work performed
tried to identify why this occurred and opportunities to reduce the average days to appointment.
2.2 OPERATIONAL FINDINGS
Customers from outside County being provided services.
Over 400 non-Deschutes County veterans have been identified as being served in Deschutes County. This
represents some 11% or more of appointments which reduces opportunities for Deschutes County customers
to be served.
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 8
Chart VI –
Composition of
services by
customer location
Providing services to non-Deschutes veterans is a likely contributor to longer await times for getting an
appointment. Deschutes County Veterans’ Services Office is only supposed to deal with Deschutes County
veterans.
Deschutes County does not have a policy for dealing with non -Deschutes County veterans seeking service.
The ODVA is responsible for veterans who do not want service from their own county.
Some reasons for out-of-area veterans using Deschutes include:
veterans that have moved out of the area and continue to want service;
veterans that are not happy with their local service and hear about Deschutes County’s service; and
veterans whose proximity to Deschutes County makes it easier to come to Deschutes.
It is recommended for a policy to be developed by Deschutes County Veterans’ Services Office to
address whether and the extent of services provided to non-Deschutes County veterans. This policy
should consider moving towards a situation of limiting “out of county” veterans and referring them to
ODVA when there is a backlog to see new Deschutes County veterans.
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 9
Chart VII –
Trend in new
client visits per
day
Long wait times for appointments (60-90 days) being experienced by new clients.
The Veterans’ Services Office has been experiencing a decline in their ability to handle new client visits.
Activity appears to be down in new client visits primarily from turnover in positions. Transitionally as new
VSO’s came on, the numbers of vet visits per day would go up. Currently, the new assistant VSO is just
starting to take on new clients and is starting to impact visits. But overall numbers have declined since two
years ago. Activity for the primary VSO over this time period is pretty consistent.
Source: Veterans’ Service software
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 10
New clients, for this audit, were defined as new clients and clients with new claims. This was determined
through the identified legal forms recorded in the software system. This should provide a reasonable proxy
for actual activity within the office. The above averages were determined from counts of clients seen for
those days where there was any activity as registered into the veterans’ services software. New clients are
currently being scheduled out 60-90 days and initially provided opportunities to schedule on two days of the
week. Even with that, they have currently only been able to get on average around 2 new veteran visits per
day. Their past history shows they can easily process 3.5-4, nearly twice the current level.
Some issues with how the appointmenting works is that clients are pushed out by only having opportunities
for meetings on two primary days and during 9 am-3:30 pm. Only after they fill those slots do they open up
8am. The other two work days focus on follow-ups and shorter appointments, but can be used for new clients
on an as needed basis.
By deferring some new client appointments out 60-90 days, it is possible that some veterans’ claims are not
being requested up front and they may lose out on some of those benefits. When possible the Office does
place a temporary claim prior to appointment. Benefits can only start to accrue after the claim has been
made. The audit did not attempt to quantify how much may be lost.
One significant reason for the decline in new clients being seen is the turnover of personnel. The Veterans’
Services Office has been in the process of training a new assistant VSO. The sooner the new staff become
fully trained and can contribute to services; the sooner there will be an impact to the backlog. The Office’s
change to only scheduling new clients on Monday and Tuesday and providing all-day walk-in clinics for non-
new clients has possibly contributed. The Office indicates they have been experiencing more requests for
service from the effects of the recession and are expecting more from the veterans coming back home from
recent conflicts.
It is recommended for the Veterans’ Services Office to develop a plan to reduce backlog for new client
appointments to an acceptable level.
Some suggested ideas for reducing the backlog in new client appointments, could include:
Providing more opportunities for new client appointments. This could include:
o extending new client appointment options to all work days,
o doing away with walk-in appointments and doing fill-in for cancellations or no-shows, and
o establishing performance measures that will monitor work towards improving new client visits.
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 11
Triaging new clients and making sure new VSO comes on quickly and has opportunities to increase
caseload.
Seeking temporary additional assistance from an available ODVA certified VSO to help bring down the
backlog faster. {It is not clear whether one may be available}
Veterans’ Service software not fully utilized.
The Office has not been utilizing or completing some of the information in the ir computerized software
system. That includes identifying and noting office visits, outgoing mail, and other communications. The
software is used to aid in preparing forms and documentation needed to manage cases. The Oregon
Department of Veterans’ Administration (ODVA) also oversight responsibility for the program and assists
many County offices.
In the absence of utilizing the software to its fullest, Deschutes County Veterans’ Services is not activating
the counts for some of the services they are performing. In addition they are creating additional manual
processes that can more easily be centralized through the system. This jeopardizes a potential trail of
activity in the records for the client. It also does not indicate the extent of work being performed.
A timesheet study performed indicated a number of meetings that did not translate into activity with a client in
the system. The software system had 41% additional vet names not recorded in time spent. Timesheets had
activity of some 47% not accounted for in the software. At best, this is some indication of the complexity and
number of clients touched but for which there is no way to easily capture the activity.
Staff only used some of the software system and have not always seen the merits of utilizing some of the
features within the system. There may be some question as to the benefits of adding some information to the
software system.
It is recommended for the Office to consider more fully utilizing the software for communicating and
operating with client information.
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 12
Performance measures could be improved.
The current performance measures utilized by the Office are focused on output measures – as in visits and
dollar awards. The audit indicated not only do the award values not fully recognize the monies that are being
collected, but not how well they are doing assisting clients. Outcome measures can sometimes be more
meaningful. As performed during the audit, the customer survey of customer satisfaction is a good example
of a relevant measure. In addition, many of the processes for collecting information on these measures are
manual. More could come from the software system if data was entered. Some issues were noted with
some of the software reports, based on insufficient entered data or incorrect data. Reports appear to be
understating awards being collected.
The Office provides much of this information to the State. Yet, it is unclear how this information is used and
reports are not provided.
In the absence of adequate performance measures and an efficient means to collect the data, it may be
difficult to assess whether the Office is operating effectively.
The Office has been having some difficulty getting some of the information of these measures from the
computer system they use.
It is recommended for the Office to consider additional performance measures that are useful in
addressing the level and quality of services being performed.
It is recommended for the Office to seek methods for collecting their information for measures from
their computerized systems, when possible.
2.3 FISCAL AND BUDGETING OBSERVATIONS
Remote services cost effectiveness in question.
Currently, remote services are provided several times a month (on an as needed basis) in Redmond, LaPine
and Sisters. The Office incurs additional costs for having Redmond and LaPine locations, as they are
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 13
charged for the associated costs for occupying (even briefly) County provided space. An estimate of the
costs per day for activity by location indicates costs are higher in the remote locations due to the additional
building charges incurred. Depending on how you calculate the additional cost per day, additional costs
incurred for utilizing the remote sites are around $10-12 thousand.
Over the last number of years the Veterans’ Services Office thought outreach to the cities of Redmond,
LaPine and Sisters would be used by veterans. The VSO has noticed that a lot of veterans still want to come
to the Bend office. The Office is no longer sure that the cost is worth the benefit, since many of the non-Bend
veterans are choosing to make the trip to Bend for services.
The Office is contemplating whether the potential cost savings from not having the remote sites would help
increase the FTE for their less than full time assistant VSO. The estimated cost savings would substantially
fund the remaining additional .25 FTE cost (~ 25% short). The additional .25 FTE might further contribute to
providing the needed services.
In addition to costs savings, there would likely also be time savings now being used to travel back and forth
from these remote locations.
It is recommended for the County to consider whether the use of the remote locations is still
warranted and evaluate whether the additional .25 FTE in staffing could have a greater impact on
service levels.
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 14
3.
MANAGEMENT
RESPONSE
Veterans’
Services Office –
Keith
Macnamara,
Veterans’
Services Officer
To: David Givans February 13, 2013
CC: Erik Kropp
From: Keith Macnamara, CVSO
Subject: Management response to Audit Report
What was found:
I agree that new appointments are primarily scheduled over two days; however it should be noted that follow -
up appointments are seen during the remainder of the work week and walk-ins are seen when ever possible.
This is also a recent change to determine the best schedule to see as many clients as possible within the
work week.
Background on Veterans’ Services:
I agree that our reports to the state show that we have recovered over $2 million dollars in benefits for our
clients. This number only represents the amount of retro award payments that our clients receive during the
specific reporting period. It is shown by the Department of Veterans Affairs FY11 Geographic distribution of
VA expenditures (GDX) that the VA spends over $68 million dollars in Deschutes County and of that $68
million over $39 million is directly paid for compensation and pension benefits. This is a direct reflectio n of the
amount of carryover benefits that we are unable to track from the time the client receives their benefits to
current report. Compensation and pension benefits are approximately 90% of what we request most for our
clients with the remaining approximately 10% for educational, medical, burial benefits, and other
miscellaneous benefits.
It states in the audit report that “they are highly trained employees on the specific legal and process
requirements from the Oregon Department of Veterans Affairs (ODVA)”. This should also include the legal
processing requirements from the Federal Department of Veterans Affairs.
Customer survey results were positive:
I highly agree with the survey results with customer satisfaction. This also shows that the numbers of clients
for follow-up appointments are higher than for other type of appointments. This is a direct result showing that
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 15
MANAGEMENT
RESPONSE
(continued)
for each new client we have they also have numerous follow-up appointments that responses are both time
sensitive and required.
Operational finding:
I fully agree with the auditor’s opinion that providing services to non-Deschutes County veterans is a likely
contributor to longer wait times for getting an appointment. I also agree with the fact that we do not have a
policy for dealing with non-Deschutes County veterans, however we have begun to refer veterans from
adjacent counties to see their respective service offices for their benefits.
Long wait times for appointments (60-90 days) being experienced by new clients:
I do agree that our new client’s average wait time per day has gone up especially with the turnover of
assistant veterans service officers. Our goal is to see, on average, 6 – 8 new clients per day when the current
new assistant is completely trained and is comfortable with all aspects of the position. With the requested
increase from .75 to 1.0 FTE for the assistant position this could be accomplished much more rapidly and
thereby decreasing new appointment wait times.
Paragraph 2: The report states that “they have currently only been able to get on average around 2 new
veterans visits per day”. This average is based on a work week and not just on the 2 days that we see new
clients. We currently see 4 new clients per day on the 2 days we are scheduled to see new clients with walk-
ins being seen if time permits.
Paragraph 3: The report states that we have issues with appointmenting and that clients are pushed out by
only having opportunities for meetings on two primary days during 9 am-3:30 pm. And only after that we fill up
other time slots at 8 am. I do agree with this statement; however the other time is required for claims
development, which includes evidence gathering and completing the write up claim justification cover sheet.
Depending on the complexity of the claim this could take any where from 1-3 hours per claim, and could take
even longer.
Paragraph 4: I do agree with the auditor’s report showing that some new clients are missing out on potential
benefits due to the backlog of services. As stated we do try and submit informal claims to preserve the date of
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 16
MANAGEMENT
RESPONSE
(continued)
claim to prevent any loss, but in some cases this is not possible.
Paragraph 5: I agree with the auditors reasoning that one significant reason for the decline in new clients
being seen is the turn over of personnel. I am confident that the new assistant hired in August will prove to be
a long standing and highly successful asset to the department. I also agree that we recently began only
seeing new clients only on Monday and Tuesdays, however we see follow-up appointments on the other work
days and see walk-ins as time permits or if we have a cancelation or no-show. I do not agree that this has
contributed to the back log since this has been a recent change and an attempt to find the best possible
schedule to deplete the backlog.
Veterans Service software not fully utilized.
I do agree with the auditor that we are not utilizing the veteran’s software to the fullest extent. The reason for
this is the time that is required to completely enter all the necessary information. It is likely that a great deal of
recoveries is not being accounted for, and also not being credited to our quarterly reports to the state. I agree
with the recommendations to more fully utilize the software for communicating, operating, and documenting
the amount of money brought into the county.
Performance measures could be improved.
I agree that the current performance measures do not fully demonstrate the potential of the d epartment.
Specifically “the audit indicated not only does the award values not fully recognize the monies that are being
collected”. This is evident from the VA geographical information stating that over $39 million dollars is paid
directly to the veterans of Deschutes County for compensation and pension benefits in FY 2011which is the
most current data available. Although this extent of data would be impossible to fully track we, with added
support, could do a better job accounting for what benefits our clients are receiving.
Remote services cost effectiveness in question:
I highly agree that closing the outlying offices in LaPine and Redmond would allow for improved time effective
services to our clients. These offices are only staffed 2 days per month per office and we are currently paying
for services as if staffed full time. The service officers would still be able to conduct outreach to these
Veterans’ Services - Effectiveness report #12/13-1 February 2013
Page 17
locations as needed and at locations that have been offered to us from other veterans organizations free of
charge for their use. The time savings, the elimination of building expenses and equipment expenses would
greatly impact the added cost of increasing our current .75 FTE to a 1.0 FTE employee. The cost of operating
these offices closely approximates the cost of adding the .25 FTE and would greatly allow for added benefits
to our veterans and their family members.
{End of Report}
Please take a survey on this report by clicking on the attached link:
http://www.surveymonkey.com/s/Veterans_Services_Effectiveness_report_1213_1