Steven Broadbent Steven Broadbent

Post tenebras lux is the Calvinist motto which is indelibly carved into the stonework of the Reformation wall in Geneva; Calvin’s adopted city.  It translates as ‘Light After Darkness’ and seems apt to convey my personal feelings following three days of attendance at the Fifth International Conference on Establishment Surveys, held in Geneva 20-23 June 2016.

From the outset I should state that EDI work across sectors and have a broad range of survey and project experience which, whilst covering establishments and institutions such as farms, health centres and schools, have the majority of our history in socio-economic research and a proud reputation in conducting household level surveys.

That said why do I feel like the ‘darkness’ is being left behind?  There are several reasons:

Sampling and unit identification. Determining the sample from a wide variety of establishments is a challenge and hence much reference to ‘domains’ was made to help quantify this further – essentially the stratification of the population. However even dividing the population into unbiased domains doesn’t tackle the issue of the unit of observation.  It appears as though establishment surveys have utilised larger units (establishments) over time but are increasingly looking at ME to supplement the larger sized populations.

I don’t mean me personally but to Micro Enterprises – often owner/manager businesses or certainly those with fewer than five employees.  In aggregate these are important when considering total employment but a ME is, to my mind, not dissimilar to a Household and the owner/manager effectively the Head of the Household.  Consequently this strikes me as one of the keenest areas of cross-over been social and enterprise surveys.

Use of Technology. I was delighted to provide a software demo on ‘Enhancing survey data quality with surveybe’ as a parallel session during the first full morning of the conference.  What became apparent during the conference was the lack of deployment of electronic data collection methods until recent times.  Many establishment surveys seem to continue to rely upon paper-based surveys.  Some presenters explained that they have introduced mixed-mode methodologies – primarily CATI and CAWI –  but to the apparent exclusion of CAPI due to cost.

Whilst embracing technology to move beyond paper is certainly a step in the right direction on several occasions, in different sessions, the phrase ‘electronic’ was taken to mean simply web-based surveys.  Many of the projects and agencies who presented are moving exclusively to web based methods which, for all good reasons, is an understandable move but I fear that rather than coming fully into the ‘light’ from the ‘darkness’ of paper-based methods these decisions will render data collection for establishment surveys ‘in the shade’ where unseen pockets of data will remain undetected as on-line collection, by its’ nature, will bias the sample and possibly overlook respondents due to a lack of connectivity or technical competence.

Embracing technology in a broader, mixed fashion, through deployment of different Computer Assisted Interviewing (CAI) methodologies, would help overcome this.

Evidence.  Establishment surveys continue to rely heavily on self completion and, even where technology has been deployed via phones and the internet, there are still missing evidential aspects that weaken the strength of the dataset.  For example the US Bureau of Labour Statistics have recognised this when assessing jobs for their cognitive, physical and environmental demands to make decisions on disability claimants in social security.  The example given by Mr. Doyle, Assistant Commissioner for Compensation Levels and Trends, was in the assessment of ‘reaching up above your head’, where phone interviews or a self-completion via the web have exactly the – you can’t readily determine how far someone can reach above their head!  The solution that they are trialling?  Interview them face-to-face.

Similarly occupational health and safety surveys, funded at EU level, used CATI to ask about Health & Safety practices and yet little evidence was gathered to indicate where stated and actual practices differ.  Therefore if the respondent merely listed the known statutory requirements of the territory in their response then the actual practices remained ‘in the shade’ as the potential for misrepresenting the level of H&S provision by employers within the EU.

Administrative data and classification. Establishments, like households, do not operate in isolation and, in order to function, establishments require employees.  Household members, in many cases, rely on employment for their finances – although this is often not the case for subsistence households in the developing world.  As a consequence establishment surveys would benefit from the utilisation of other, often government collated data, such as tax records which would support declarations of establishments across a range of statistics from profits, R&D expenditure, tax payments in all its forms and so on.

Here a fragility of evidence increases further as regulatory and legal restrictions prevent use of such data.  This wealth of data, even if it was accessible, is then further corrupted by a lack of harmonisation of classification.  Therefore a body of work became evident, especially from national statistical offices, to improve this situation with the intent of permitting like-for-like comparisons and analysis over time.

Marketplaces are dynamic with new, previously non-existent jobs, industries and businesses being created – therefore the classification task is one that may never end and so governments need to become agile in their response.  Thankfully the US BLS are mindful that their catalogue of job roles, previously updated annually, was last updated in over twenty years ago.  Imagine how many internet related jobs are being forced into outmoded, outdated and wholly inappropriate classifications!?!

Respondents and non-response. Commentary was made by several presenters and discussants over the issue of non-response.  The primary methodologies deployed on establishment surveys result in a high level of non-response, if the presentations I witnessed are to be believed.  Some researchers reported a correlation between establishment size and increasing propensity for non-response.  This in turn causes difficulty in the weighting and representativeness of the sample and, as most participation is voluntary little could be done about this.

Where follow-up protocols were followed then they were often conducted a long time after the initial engagement, exposing the study to the difficulties of forgetfulness, employee turnover and lack of organisational knowledge and authority by the respondent.

In point#1 above I make mention of the difficulties of establishing units of observation; this is further compounded by a lack of certainty over who is responding on behalf of the establishment.  Has data been collected from the most appropriate and relevant person in the organisation?  If not then one may identify the risk that this may cause significant gaps in the data set through missed or limited data to be recorded.

Maximising data quality is surely the goal of all researchers no matter their area of focus and specialisation and, despite some of my concerns outlined above, this conference has shown that those interested in this area are desperate to provide the best possible data.  One could sense the breaking of a new dawn with every screen of further work, planned projects and the adoption of new methodologies to deliver positive results.  It is certainly time for the vast array of establishment data to take centre stage in discourse at all levels of society – we saw the potential impact of this throughout the week in all sectors and, such work is no easy task.  I congratulate those engaged in such work and hope that they enjoy being bathed in the light that good quality data provides!