Outline of the reference costs audit methodology
Approach
To form a view on the accuracy of a trust's reference costs submission our auditors investigated at four different levels within each organisation. They looked at:
- the organisational arrangements for ensuring the accuracy of submissions;
- the accuracy of the total cost quantum used;
- the quality of activity reporting by service; and
- the accuracy of individual unit costs within the organisation.
Materiality
Throughout all our testing we used the principle of materiality to drive investigation and to guide judgements. We viewed anything that caused a movement of more than 1 per cent of the total cost quantum as being a material error. For an average trust, 1 per cent would be around £2.23 million. We used other thresholds when targeting investigations at service or individual unit cost level. This approach allowed us to understand the implications of issues we identified, and to target our investigations on areas of most benefit.
Arrangements
Auditors assessed each trust's arrangements for securing good quality reference costs data against four themes. This enabled them to draw a rounded judgement on the accuracy of a trust's reference costs submission. It also helped them understand the processes behind producing the activity data and calculating the costs of activity. The four themes were as follows:
- Engagement - there is senior level engagement in, and support for, the data quality of reference costing data.
- Activity reporting - there are arrangements to support recording all relevant activity “right first time” at final submission.
- Approach to costing - there are arrangements to ensure a robust approach to costing.
- Data capture systems - the organisation's IT systems produce robust activity and costing information.
Our auditors assessed whether each theme was:
- below minimum requirements - inadequate performance;
- only at minimum requirements - adequate performance;
- consistently above minimum requirements - performing well; or
- well above minimum requirements - performing strongly.
We made judgements on arrangements once all the other parts of the audit were completed, as the review of activity and costing data often highlighted issues not identified from a high-level review of arrangements.
Total costs
Trusts must correctly calculate three things to ensure the individual unit costs are accurate. They are:
- total costs included;
- total activity; and
- allocation of total costs to activities.
Trusts should compile reference costs data using a 'full absorption' costing method. This is where a trust includes all the relevant costs associated with relevant hospital activities. Simply put, the total costs included in reference costs should reflect the total cost of running the hospital in that year. However, trusts can legitimately exclude some costs and activities, and extra adjustments can occur based on guidance from the Department of Health.
By reconciling the total cost quantum to the accounts and other sources, reviewing the adjustments made, and looking into any variances, we are able to take a view on whether the total cost included in the reference costs submission was correct.
Individual unit costs
It would take more time than available during an audit to check costing processes and compliance with the costing manual guidance, so it is not practicable for auditors to review the accuracy of all data items. Instead, we used comparative analysis and comparing data to other sources to identify local variances. We targeted the audit of cost and activity data on these areas. The analytical tools used to do this are available online in our National Benchmarker.
To do this we looked for inconsistencies in activity reporting by service, and in cost allocation by individual healthcare resource groups (HRG), by assessing the following areas:
- whether the reported reference costs activity is in line with Hospital Episode Statistics (HES);
- whether any reported reference costs activity data is greater than expected for the size of trust;
- whether there are any extreme unit costs (HRGs under £5 or over £50,000); and
- whether there are any material variances from what was expected, for individual unit costs.
Method for identifying trusts for clinical coding and outpatient audits
We focused our inpatient clinical coding audits and outpatient audits at the most poorly performing 20 per cent of trusts in each area. The trusts audited were selected in two ways:
- For inpatients, ranking trusts using the HRG assignment error rate from the last three years' PbR assurance audits, weighted to 2009/10. We removed any outliers caused by exceptional circumstance in any one year, and trusts that significantly improved in 2009/10.
- For outpatients, ranking the trust by the percentage of attendances with one or more errors, focusing on trusts whose arrangements were below minimum requirements.
We encouraged PCTs to choose the areas to audit, although one speciality had to be the same as one audited in a previous year to track improvement.
Method for identifying independent sector (IS) hospitals for audit
We chose the 30 IS hospitals for audit by:
- identifying the main providers of NHS activity covered by PbR and selecting the hospital managed by each of these suppliers with the largest activity levels (based on secondary uses service (SUS) submissions); and
- ranking the remaining independent sector hospitals by activity under PbR.
Method for calculating the financial impact of coding errors
To understand the financial impact we apply PbR payment rules to pre- and post-audit coded information using the payment grouper. The grouper uses the same rules as the SUS to derive full elective or non-elective PbR prices, including short-stay adjustments, excess bed days, and specialist top-ups.
A net change is the actual impact of changes on the sample. Positive and negative values are added to together and then applied to the audit sample to understand the actual impact of errors on payment.
A gross change shows the total financial value of the errors identified. The polarity of the changes for each error is ignored (positive or negative), and the total value of the changes is summed to contextualise the impact of data quality on the payment system.
A full description of technical terms used in the report is provided in the glossary.