BOB: Business Objects Board
Not endorsed by or affiliated with SAP

Register | Login 

Want to sponsor BOB? 
Want to sponsor BOB? (Opens a new window)  

General Notice: No events within the next 45 days.

Flagging of Non-Robust Data


 
Search this topic... | Search General Discussion... | Search Box
Register or Login to Post    Forum Index -> General Discussion  Previous TopicPrint TopicNext Topic
Author Message
rds3wave
Forum Member
Forum Member



Joined: 19 Jul 2011
ASUG Icon
Posts: 25
Location: Chelsea, MI



PostPosted: Fri Aug 04, 2017 9:00 am 
Post subject: Flagging of Non-Robust Data

In healthcare, we depend on accurate charting, and accurate translation by billing coders, to provide visit data to our warehouse. Some charting is inadequately done...and thus some fields in some fact tables are not reliable or robust. (For example, an identifier field to indicate where this was patient sent after their visit was complete -- home, admitted, morgue, etc....--will not produce accurate aggregates if not always filled in.) Are there standard techniques for marking or flagging certain DW fields as "not to be trusted"?

I am mostly interested, at the moment, in the final step. If we know that the "disposition" field only gets populated by the clinic 10% of the time (we might determine that after doing profiling, and doing some comparisons to real data), what's the best way to communicate that meta-information to users, other than, say, adding an adjacent column called something like "level of accuracy of the Disposition column" and either putting "0.10" in there, or "N" for "not trustable at this time". Seems like there might be a more elegant solution.

Thanks for ideas --
Back to top
ABILtd
Forum Enthusiast
Forum Enthusiast



Joined: 08 Feb 2006
ASUG Icon
Posts: 1742


flag
PostPosted: Thu Aug 10, 2017 5:00 am 
Post subject: Re: Flagging of Non-Robust Data

I`ve done stuff for social care reporting in the past, where specific fields, if not filled won`t allow you to accurately calculate KPI`s and thus can cost money if the KPI`s are down.

Effectively all I did was build a few data quality reports from the source systems, look for nulls (or in some cases bad patterns) and then flag each row with a data quality score. These reports where built at the lowest level of data, and could identify individual people who had DQ issues (or teams). It least then it can be addressed at source, where DQ issues should be addressed. More often than not it`s a training issue, or mis-understanding, but sometimes just laziness!

_________________
BI and Analytics Presales Consultant
------------------------------------------------------
BOXI R1, R2, R3, R3.1, R4.1, R4.2, CE 8, 8.5, 9, 10, Crystal Reports, Lumira, PowerBI, Tableau, JasperServer, iReport, LogiAnalytics, BIRST, Qlikview, Xcelcius, Netezza, T-SQL, PLSQL, DTS, SSIS, BODI, BODS, Kimble etc....
------------------------------------------------------
Back to top
rds3wave
Forum Member
Forum Member



Joined: 19 Jul 2011
ASUG Icon
Posts: 25
Location: Chelsea, MI



PostPosted: Thu Aug 10, 2017 7:16 am 
Post subject: Re: Flagging of Non-Robust Data

Nice ideas, thanks much. It's spurred me to think about this from a couple new angles.
Back to top
Maddye
Principal Member
Principal Member



Joined: 09 Jan 2009

Posts: 126
Location: Cambridge


flag
PostPosted: Thu Aug 10, 2017 9:49 am 
Post subject: Re: Flagging of Non-Robust Data

I do adult social care reporting, and most reports I build have DQ built in. It's usually stuff like missing demographic data which prevents the row from being counted in various statutory returns, but I also include business process errors like 'you've ended this episode with an outcome indicating further work is to be done, but you've not recorded any further work 28+ days later'.

It sounds like you've got a similar issue. You probably also need to talk to the DBA about making certain fields mandatory.

_________________
Next available for contract work November 2017
Back to top
ABILtd
Forum Enthusiast
Forum Enthusiast



Joined: 08 Feb 2006
ASUG Icon
Posts: 1742


flag
PostPosted: Fri Aug 11, 2017 10:26 am 
Post subject: Re: Flagging of Non-Robust Data

Maddye wrote:
I do adult social care reporting, and most reports I build have DQ built in. It's usually stuff like missing demographic data which prevents the row from being counted in various statutory returns, but I also include business process errors like 'you've ended this episode with an outcome indicating further work is to be done, but you've not recorded any further work 28+ days later'.

It sounds like you've got a similar issue. You probably also need to talk to the DBA about making certain fields mandatory.


Exactly! Painful isn`t it!

I think I kept my DQ reports separate as I used to send them out on a weekly basis as a 'name and shame' mechanism.

_________________
BI and Analytics Presales Consultant
------------------------------------------------------
BOXI R1, R2, R3, R3.1, R4.1, R4.2, CE 8, 8.5, 9, 10, Crystal Reports, Lumira, PowerBI, Tableau, JasperServer, iReport, LogiAnalytics, BIRST, Qlikview, Xcelcius, Netezza, T-SQL, PLSQL, DTS, SSIS, BODI, BODS, Kimble etc....
------------------------------------------------------
Back to top
Display posts from previous:   
Register or Login to Post    Forum Index -> General Discussion  Previous TopicPrint TopicNext Topic
Page 1 of 1 All times are GMT - 5 Hours
 
Jump to:  

Index | About | FAQ | RAG | Privacy | Search |  Register |  Login 

Get community updates via Twitter:

Not endorsed by or affiliated with SAP
Powered by phpBB © phpBB Group
Generated in 0.0162 seconds using 18 queries. (SQL 0.0083 Parse 0.0003 Other 0.0077)
CCBot/2.0 (http://commoncrawl.org/faq/)
Hosted by ForumTopics.com | Terms of Service
phpBB Customizations by the phpBBDoctor.com
Shameless plug for MomentsOfLight.com Moments of Light Logo