Keith E Torkelson - Technology Patient Portal Study – Quality & Value - Audit Results – Report-01 - Also – AVEY - Buster

Keith E Torkelson, MS

Technology Patient Portal Study

 

Quality & Value - Audit Results

Report-01 - Also – AVEY - Buster

 

Universal Care – Brand New Day

Aerial - Cerecons’ Patient Portal Audit & Evaluation

By Mentalation Solutions Group (MSG, 2016) & Keith E Torkelson, MS

 

Patient Portal Audit Score (PPAS) Report

	Date Last Scored	Assessment	Calculation	Result
	-
	20160827-SAT Items Scored = 13	Patient Portal Content Audit Score [PP-CAS]	2 PASS/13 Items X 100 =	15%
	20160823-TU Items Scored = 22	Patient Portal Quality Audit Score (Errors) [PP-QAS]	2 PASS/22 Items X 100 =	9%

20160827 - Patient Portal Audit Report Card (No Control Available)

 

Screen Shot of Brand New Day - Cerecons Dashboard (20160823-TU)

 

Abstract – Executive Summary

We here at Mentalation Solutions Group (MSG) have performed both a 22-point Quality Audit and a 13-point Content Audit of our program’s Patient Portal (Cerecons – Aerial).  We have no control patient portal for this study.  We were limited to evaluating only our own record (n=1).  We found that out of a sampling of four (4) members in our program, Brand New Day (BND), that only we were referred to using the patient portal.  After thoroughly reviewing the content in our record we determined that our record’s Patient Portal Content Audit Score (PP-CAS) as of 20160827 is 15 percent (%).  After reviewing the quality of our record we determined that our record’s Patient Portal Quality Audit Score (PP-QAS) as of 20160823 is 9 percent (%).  These values are incredibly terrible.

 

 

Follow Up Action Items

It is very likely that the errors are split between programming and coder errors.  We here at MSG would not recommend our portal content for a Health Information Exchange (HIE).  We are sharing our findings with the Mental Health Services Act – Orange County – Technological Needs Group (TAC) with the hope that it will help them avoid any of the pitfalls we pointed out in this report.  We did find a puzzling glitch with entries under Physicians Recently Visited.  We have no recollection for visiting four (4) of the practitioners listed.  Lastly, we offer Brand New Day (BND) portal administration the chance to have us correct our own record.  Due to issues with confidentiality, which we have run across in the past, we know it is unlikely we can correct any but our own Brand New Day – Cerecons patient portal record.

Round-01 Documents – Provider Series-01-

Worksheet	Tech_Patient_Portal_Study_16090202_WS-01
Assessment/Audit Results	Tech_Patient_Portal_Study_16080402_Assess [THIS]
Brief Appendix	Tech_Patient_Portal_Study_16080402_Append-Brief

My Medications - A First Impression – Screen Shot

Figure – My Medications Screen Shot – Marked Up

Error Types – My Medications (3/22 of total errors found = Audit Error Share)

	-
#	Error Type	Example	Better if
	-
1	Incorrect Value & Impossible Value	60000 for Quantity	60
2	Date Format	Oct 17 2015	20151017
3	Missing Header	Date Header Omitted	Included

Table – Medication Domain – Errors – Wrong and Impossible Values

Metadata Specs - Range of Medication Data (2012-2016) Range of Errors (2014-2016)

Likelihood of Getting the Medications’ Section (Domain) Right

Due to the nature of both coding and programming errors: It is likely these errors occur in other clients’ patient portal records.  The chance that these errors can be corrected is very small.  We recommend deletion and recoding with an improved coding quality framework in place. (Level of Error(s) when extrapolated across all clients = Crash Worthy)

Outline

Pages Predicted for This Brief Report (25)

It is good that we here at MSG predicted a limit to our brief report because at one point our brief report exceeded fifty (50) pages.  To keep this report brief we separate our appendices PRN.  For example: Introduction – Brief [20160830-TU SEPARATED] & Seven Issues (No PASSES) for Fleshing [SEPARATE]. 

Report Content Inventory (For Brief Report)

	Section - Description	Keyword	Page(s)
	Title Page - Blog Post Title Result Up Front (RUF) - Audit		1
	Abstract – Executive Summary		3
	My Medications - A First Impression – Screen Shot		4
	[THIS OUTLINE]		5
	Audits for [Content] and Quality		6-7
	Audits for Content and Quality Patient Portal Audit Score (PP-AS) Report Card		8
	Acronyms for Quality Audit		8
	Audits for Content and [Quality]		9-10
	My Doctors Tab > Physicians Recently Visited (Concern & Most Puzzling) Confirmation Research Results		11-12
	1-Syntax for Dates - Format Quality &Consistency		13
	Miscellaneous Lose Ends		14
	Last Minute Moves & Summary of Removals		14
	Discussion Items		14

Table - Report Content Inventory (For Brief Data Report – TAC Version – 20160902-F)

 

 

Audits for [Content] and Quality

This is the section for our: Patient Portal Audit Scores Report (PP-ASR).  In this section we address both Patient Portal Content and Quality.  Using the Dashboard as a framework we evaluated (audited) our Patient Portal for Member and Clinical elements.  Our cutoff for a terrible interpretation is fifty (50) percent or less.  This content area framework was derived from the patient portal itself.  Most of the domains come from the Dashboard Level (Primary Level).  In a few instances we drilled down into Secondary Levels.  We reviewed the material in the following table last on 20160827-SAT.  This assessment yields a Patient Portal Content Audit Score [PP-CAS]. 

Audits for [Content] and Quality - Data

	-
##	Patient Portal Element or Tab	Cerecons Error - Test Outcomes	Note Regarding Fail
01	My Info - Demographics	PASS
02	My Doctors	Fail	Unknown Doctors
03	My (Member) alerts	Fail	Alerts out of date by 2 years
04	Send an Email Non-Emergency	20160701-20160730 Fail	Close effort after 2 attempts Conclusion - Unsupported
05	My Links > Email	Fail	Tested Email Option
06	My Medications	Fail	Numerical Errors Errors with dates
07	My Recent Allergies	Fail
08	My Visits	Fail	Not enough info to determine if I really saw all of them listed
09	My Recent Labs	Fail	Incomplete No Reference Ranges
10	My Recent Referrals	PASS
11	My Immunizations	Fail	Not reported yet I have had several of them
12	My Medication Reviews	Fail	We do it every 3 months – not reported
13	Assessments	Fail	Submitted  - Yet never reported
	Content Audit Score (CAS) =	2 PASS/13 Items Scored = 15%	It Fails on any grading curve

Table – Patient Portal Content Audit Results (MSG, 20160827)

Also see: [PENDING]

Appendix Brief – Login Information

Appendix – Printing the Dashboard

Audits for Content and Quality - Cumulative Report Card 

Patient Portal Audit Scores (PP-AS) Report Card

	Date Last Scored	Assessment	Calculation	Result
	-
	20160827-SAT Items Scored = 13	Patient Portal Content Audit Score [PP-CAS]	2 PASS/13 Items X 100 =	15%
	20160823-TU Items Scored = 22	Patient Portal Quality Audit Score (Errors) [PP-QAS]	2 PASS/22 Items X 100 =	9%

Table - 20160827 Patient Portal Audits – Cumulative Report Card - Patient Portal Audit Report Card (No Control Available) 

Acronyms for Quality Audit (Brief)

	-
AcroCode	AcroCode	Note
	-
A-SEP	Addressed Separately
IFF	Issues For Fleshing
NAF	No Assessment Found
NAR	No Assessment Reported
	-

Table – AcroCodes (MSG) for Patient Portal Quality Audits Results (PP-QAR)

Scoring Method(s) – PASS/Fail

Basically, we use a PASS/Fail method to score the items in both the content and quality assessments.  All items received a Fail unless as PASS is scored.  As addressed earlier we have no true Controls for this study.  Blank scoring forms are included in our brief appendix handouts.  The total number of items for Content is thirteen (13), whereas the total number of items for Quality is twenty-two (22).

Audits for Content and [Quality]

The next couple of pages address the Quality of our Patient Portal.  Our summary findings for Patient Portal Audit Scores Report (PP-ASR) were presented in the table above.  Below are the details about our Patient Portal quality assessment.  The tables include our Patient Portal Audit Data.  After working on this formal study since winter quarter of 2016: We now consider our organization (MSG) educated about Patient Portals.  There are two parts (tables) for our Quality assessment. 

Page 1 of 2 – Patient Portal Quality Audit - Data 

	-
Item #	Error Type	Description	Sample Incidence	Correct [PASSES]
	-
01	1-Date	Format Quality & Consistency		[A-SEP] - IFF
02	Activity Counter(s)		Absent
03	Anonymous Coders (QC)		Global
04	Assessments	Evaluated	NAF
05	Assessments	Reported	NAR
06	Client Input	Updating Fields	My Visits
07	Color	Duplication	Headers
08	Consistency	Across Fields	Dates
	Scored = 9 (PP-AQS Sub-score) =			# - PASSES/Items 0/8

Table - Patient Portal Quality Audit Scores (PPA-QS-9 Item) - Results – Last Scored: 20160823

Also see: [PENDING]

Appendix – Quality Audit Worksheet – 22 Item – Combined – Page 1 of 1

Audits for Content and [Quality] 

Page 2 of 2 – Patient Portal Quality Audit - Data 

Item #	Error Type	Description	Sample Incidence	Correct [PASSES]
09	Data	Incomplete	My Recent Labs My Immunizations	[A-SEP] - IFF
10	Data	Missing	My Recent Labs
11	Data	Not Up to Date	My Alerts
12	Demography			PASS
13	Detail	Lacking	My Visits
14	Errors	Copy & Paste	My Medications
15	Feature	Unsupported	My Links > Email
16	Feedback	Methods	WEAK	[A-SEP] - IFF
17	Formatting		All Over
18	Impossible	Entries	My Medications
19	Promotional & Facilities		POOR	[A-SEP] - IFF
20	Spelling			PASS
21	Use of Color		No Enhancement
22	Value	Incorrect	My Medications
	-
	Scored = 14 (PPA-QS Sub-score) =			# - PASSES/Items 2/14

Table - Patient Portal Quality Audit Scores (PPA-QS-9 Item) - Results – Last Scored: 20160823

Also see: [PENDING]

Appendix – Quality Audit Worksheet – 22 Item – Combined – Page 1 of 1

My Doctors Tab > Physicians Recently Visited (Concern & Most Puzzling)

12 of 12 Entries Captured > 8 = Correct > 4 Unknowns 

-
Unknown Physician	Specialty	Name of Real Doctor
-
Carrie Wakai	Optometrist	Ann Inman and Boyce
Shubhangi Godbole	General Practice	Chester Mojica
Zulmira Chadwick	Lic Voc Nurse	Un-needed No hits on Google
Angelita Beredo	Internal Medicine (Endocrinologist)	Un-needed
-

Table – My Doctors Tab – Physicians Recently Visited - Unknown Doctors

Action Items - My Doctors Tab > Physicians Recently Visited

-
Action Item	Progress
-
Check Internet for the Nature of Practitioners	Results on Next Page
Check “My Referrals” Tab in Patient Portal	No referrals approved for these four (4) practitioners
Review paper versions of referrals mailed to us	No referrals mailed to us
Solicit advice
Bring to the attention of Provider
Bring to the attention of CMS

My Doctors Tab > Physicians Recently Visited (Most Puzzling)

 

Confirmation Research Results

The addresses were taken from the Patient Portal

Research All Unknowns on WWW (Changed strategy)

Search Patient Portal for Details 

Doctor Carrie Wakai - Optometry

Address:  3756 Santa Rosalia Dr #100 - Los Angeles, Ca 90008

Tel: (323) 294-7517 - Fax: (323) 294-9219

Doctor Angelita Beredo - Internal Medicine

Address:  301 N Prarie Ave #301 - Inglewood, Ca 90301

Tel: Note Listed > Fax: Note Listed 

Doctor Zulmira Chadwick - Lic Voc Nurse

Address:  5455 Garden Grove Blvd 5th Fl - Westminster, Ca 92683

Tel: Note Listed > Fax: Note Listed

http://www.name-doctor.com/name-zulmira-meaning-of-zulmira-15378.html

ORIGIN: Hebrew 

Doctor Shubhangi Godbole - General Practice

Address:  12675 La Mirada Blvd #200 - La Mirada, Ca 90638

Tel: Note Listed > Fax: Note Listed

http://www.indiachildnames.com/name.aspx?name=Shubhangi

Origin: Indian: Religion: Hindu

Search for Referrals in Patient Portal

Date Range = 20160801 – 20130626 Total Referrals Recorded as of 20160825-TH: = 32

None of the Providers in Question have Referrals associated with them

 

Shortfall of Patient Portal

No method to bring up invoices (bills) with appropriate level of detail.

With regards to the mystery providers: What would you do?

-

-

-

-

-

1-Syntax for Dates - Format Quality &Consistency

Out of all of our findings we find date formatting and placement our highest priority suggestion

We here at MSG fractionate our some Issue For Fleshing (IFF).  We have referenced material about common data coding protocols.  The programmers for our Patient Portal do not appear to have referenced the same data coding standards (specifications).  They would better be served by standardizing their formats and data definitions.  The programming errors lead to problems with sorting fields such as Medication Dates. 

Variation in Date Formats as they Appear

Patient Portal Domain	Date Format	Sort Issues
-
My Medications	Oct 17 2012
My Visits	05/02/12
My Recent Labs	12/29/2015
My Alerts	2014
My Recent Referrals	08/01/2016
-

Table – Samples of Dates Formats Used in Our Patient Portal Record 

Variation in Formatting (Summary)

This whole Patient Portal package needs to be bounced back to the programmers. 

The Biggest Issues

Once again, we here at MSG find: Date Format Issues – Global: As one of the most serious problems with our client’s Patient Portal.  This utility suffers wrong and inconsistent date formatting.  This will lead to sorting problems.  This will lead to Health Information Exchange errors.  This placement of dates in a list (database) is generally the most left most field.  We recommend the following date formatting. 

YYYY-MM-DD or

YYYY/MM/DD or

YYYY MM DD or

YYYYMMDD 

Bank of America Model for Date Formats [20160901=TH: Last Minute]

 

Date in Left Most Field

No Text in Date – Text in Description

Miscellaneous Lose Ends

Discussion Items (For Next Report) 

Email Utility

Coders

Data Feeds & Sources

The Feedback Challenge

 

Summary & Recommendations (Fixes and Solutions)

“We here at MSG have some ideas about what to do next in getting BND’s Patient Portal “right”.

End of Patient Portal

Results Up Front Report (RUF-R)

Divided

 

 

MSG (2016-2017)