Purpose Service user involvement in instrument development is increasingly recognised as important, but is often not done and seldom reported. This has adverse implications for the content validity of a measure. The aim of this paper is to identify the types of items that service users felt were important to be included or excluded from a new Recovering Quality of Life measure for people with mental health difficulties. Methods Potential items were presented to service users in face-to-face structured individual interviews and focus groups. The items were primarily taken or adapted from current measures and covered themes identified from earlier qualitative work as being important to quality of life. Content and thematic analysis was undertaken to identify the types of items which were either important or unacceptable to service users. Results We identified five key themes of the types of items that service users found acceptable or unacceptable; the items should be relevant and meaningful, unambiguous, easy to answer particularly when distressed, do not cause further upset, and be non-judgemental. Importantly, this was from the perspective of the service user. Conclusions This research has underlined the importance of service users’ views on the acceptability and validity of items for use in developing a new measure. Whether or not service users favoured an item was associated with their ability or intention to respond accurately and honestly to the item which will impact on the validity and sensitivity of the measure. Keywords Quality of life · Recovery · Outcome measure · PROM · Validity · Service users · Qualitative Background There has been an increasing commitment to Patient Reported Outcome Measures (PROMs) designed to measure day-to-day health improvements that matter most to service * Janice Connell users [1, 2]. The methodological quality of PROMs has been firstname.lastname@example.org determined by the extent to which they meet the recognised properties of measurement of reliability and validity [3–5]. School of Health and Related Research, University of Sheffield, Sheffield, UK A key property of validity is the extent to which a measure captures what it is intended to measure. In the absence of a School of Health Sciences, Medical School, University of Nottingham, Nottingham, UK gold standard measure, researchers have often used indirect methods for testing validity such as factor analysis, Rasch, Centre for Psychological Services Research, University of Sheffield, Sheffield, UK or Item Response Theory (IRT). These are important in instrument development, but are insufficient on their own Department of Psychology, University of Sheffield, Sheffield, UK to fully establish the validity of an outcome measure . It is also important to measure content validity; the extent to Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK which the set of items comprehensively covers the different components of health to be measured  and face validity; The McPin Foundation, London, UK Vol.:(0123456789) 1 3 1894 Quality of Life Research (2018) 27:1893–1902 whether the items of each domain are sensible, appropriate, builds on a systematic review of qualitative research and pri- and relevant to the people who use the measure on a day- mary qualitative research involving service user interviews to-day basis . which identified the seven themes service users considered The content and face validity of many outcome measures important to quality of life referred to above [24–26]. A pool currently in use are based on the judgements of research- of potential items (n = 1597) which best represented these ers and health care professionals, with limited input from seven domains was generated from current quality of life service users [8, 9]. However, what may be regarded as a and recovery instruments and from service user interviews. good outcome by a clinician or researcher may differ from The item set contained both positively (e.g. I felt happy) what is regarded as important to service users  and only and negatively (e.g. I felt sad) worded items. These were service users can determine whether the measure captures subjected to initial sifting using a set of criteria adapted these outcomes favourably . Active input from service from those originally proposed by Streiner and Norman . users in all the development stages of a measure can improve After consideration by clinicians, researchers, and service the acceptability, relevance, and the quality of the measure users who were members of the research study’s Scientific, and related research [9, 11, 12]. Advisory, Stakeholder and Expert User Groups, the item set In recent years, there has been a change in mental health was reduced to 88 potential items for the new ‘Recovering policy from an emphasis on services that focus only on Quality of Life’ measure . The aim of this paper is to symptom reduction, towards those that take a more holis- identify the key themes of face validity which should be con- tic and positive approach of service user-defined recovery sidered when developing a quality of life instrument, using and quality of life [13–15]. This growing movement towards the ReQoL as an example. We also report on service users’ positive mental health has given rise internationally to views on acceptability and validity of an item set. recovery-oriented mental health services [16–20]. It is of particular importance that PROMs used in these services reflect the key areas that service users consider relevant Methods to recovery, rather than the traditional focus on symptom reduction . Aspects of life that are considered important A qualitative study using face-to-face structured interviews to ‘recovery’ have been shown to be consistent with those and focus groups with service users was undertaken. important to ‘quality of life’ [22, 23]. The themes service users consider important to quality of life: well-being, rela- Recruitment tionships and a sense of belonging, activity, self-perception, autonomy, hope, and physical health [24–26] are similar to There was a requirement that ReQoL be suitable for mental those regarded as important to recovery in mental health: health service users over the age of 16. Therefore, adults connectedness, hope, identity, meaning, and empowerment (aged 19–79) and young adults (aged 16–18 years) from (CHIME) . National Health Service (NHS) mental health services and The new measure Recovering Quality of Life (ReQoL) in a local charity were invited to participate. In order to obtain mental health was developed in four stages: (1) generation views from across the spectrum of mental health service and subsequent shortlisting of candidate items; (2) testing users, broad inclusion criteria were applied; the only exclu- face and content validity of shortlisted items; (3) psycho- sion concerned people experiencing acute episodes of their metric evaluation; and (4) selection if the final items which mental health condition, those not well enough to take part, involved combining the qualitative and quantitative evidence and those who could not speak English or give informed from stages 2 and 3. Importantly, service user opinion and consent. This allowed for maximum variation of mental input was utilised at all stages with a panel of six expert health problems, severity of problems, and current service service users being involved in the selection of the short- contact. listed items from the pool of candidate items through to Adult participants were recruited from four UK NHS the decisions surrounding the inclusion of the final items. Trusts providing mental health services and a UK charity Psychometric testing involved over 4000 service users com- in the north of the country. Two trusts were located in the pleting either a 60 or 40 item version of the measure before south of England and two in the north. Recruitment of young the selection of the final items. For details on all the stages adults (aged 16–18) took place from two further NHS trusts involved in the development of the measure see Keetharuth based in the Midlands and the North of England. Recruit- et al. . ment was undertaken on our behalf by healthcare staff and In this paper, we specifically report on the second of clinical studies officers within the individual trusts. Recruit- four stages which involved seeking the opinions of service ment procedures followed what was usual and practical for users on a pool of items to help inform the selection of items each individual Trust. This variously involved healthcare which would go forward for psychometric testing. This staff approaching service users on inpatient wards, when 1 3 Quality of Life Research (2018) 27:1893–1902 1895 attending therapy sessions, those who had previously Table 1 Characteristics of participants agreed to be approached for research purposes, attendees Adults Young adults at a Recovery College and at a Rehabilitation and Recov- N % N % ery Centre, and members of established PPI (Public and Gender Patient Involvement) groups. At the time of the interview, Male 22 37.3 5 29.4 demographic, care service, and diagnostic information was Female 37 62.7 12 70.6 taken. In order to obtain a representative and diverse sam- Age ple, this information was used to identify recruitment gaps 16–18 0 0 17 100 and recruitment personnel were subsequently informed and 19–29 12 20.7 0 0 asked to target these groups. 30–39 15 24.1 0 0 40–49 12 20.7 0 0 Participants 50–59 10 17.2 0 0 60–69 6 10.3 0 0 A total of 59 adult service users took part: 40 participated in 70+ 1 1.7 0 0 individual interviews; 11 attended two focus groups of seven Not indicated 3 6.8 0 0 and four participants, respectively; and four interviews took Ethnicity place with two participants together. A total of 17 young White British 46 77.6 15 88.2 adults participated: 15 participated in individual interviews Black/Black British 6 10.3 0 0 and two participants chose to be interviewed together. Inter- Asian/Asian British 2 3.4 0 0 Mixed/multiple ethnic group 2 3.4 2 11.8 views lasted between 20 min and 1 h 40 min. For informa- Other ethnic group 3 5.2 0 0 tion on the sample demographics please refer to Table 1. Employment/activity Employed 16 25.9 2 11.8 Interviews Student 2 3.4 15 88.2 Retired 4 6.9 0 0 Interviews and focus groups were conducted between Sep- House-person 2 3.4 0 0 tember 2014 and June 2015. Interviews and focus groups Not in employment 25 43.1 0 0 were undertaken at NHS sites familiar to the participants Other 8 13.8 0 0 apart from one which was on university premises. Partici- Not indicated 2 3.4 0 0 pants were provided with an information sheet prior to the Diagnosis/own view interviews. Written consent was obtained at the time of the Schizophrenia/psychosis 26 44.8 0 0 interview prior to any data collection. Participants were Depression 27 46.6 6 32.3 asked to complete a short demographics form indicating Bipolar 8 13.8 0 0 their gender, age, ethnicity, employment, education level, Anxiety disorders 21 24.4 4 23.5 Eating disorder 2 3.4 2 11.8 mental health diagnosis, and their own perception of their Personality disorder 4 6.9 0 0 mental health problem (which may or may not be the same Other 4 6.8 0 0 as the diagnosis). Interviews and focus groups with adult Not indicated 2 3.4 5 29.4 service users were conducted by three experienced qualita- Current care tive researchers, one of whom is a service user (JCo, JCa, None 3 5.2 0 0 AG). Interviews with young adults were undertaken by an General practitioner 6 10.3 1 5.9 experienced qualitative researcher and a clinician who spe- Improving access to psychological therapy 8 13.8 0 0 cialises in child and adolescent mental health (ETB). All Community mental health team 29 50.0 0 0 participants were given a £20 shopping voucher in recogni- Child and adolescent mental health services 14 82.3 tion of their time. (CAMHS) The interview process was the same for both adult and CAMHS inpatient 2 11.8 young adult service users. The items were grouped by Adult inpatient 8 13.8 0 0 Voluntary sector 3 3.4 0 0 domain, ordered, and presented one domain at a time. To Not indicated 2 3.4 0 0 reduce fatigue effect, the items were presented in a different order by starting with a different domain for each subsequent One participant identified as being transgender interview. To help establish the validity of each potential Participants provided a diagnosis and their own opinion of their item, participants were asked whether it was meaningful and problem—data here are a combination of the two with priority given relevant to their quality of life; whether it was clear, under- to the diagnosis where provided. Some participants provided more than one diagnosis/opinion of their presenting problem standable and easy to answer; and the reason they either Some participants indicated that they were receiving current care liked or disliked an item, or preferred it to another. They from more than one service provider 1 3 1896 Quality of Life Research (2018) 27:1893–1902 were also asked for their preferred item within a group or was used in conjunction with other evidences such as the pair of items thought by them to relate to a similar concept psychometrics and feedback from clinicians  to make a (e.g. between ‘I felt relaxed’ and ‘I felt calm’). Alternative decision on the final items. wordings to items were elicited if participants thought an item was important to their quality of life but unclear or difficult to answer. Results An iterative approach was undertaken with the inter- views. Adult participants were initially presented with a Analysis required that the items included in the measure set of 88 potential items. At approximately the halfway should be relevant and meaningful, be unambiguous and stage after interviews had been undertaken at two of the easy to answer when feeling distressed, did not cause fur- trusts, 12 items (which were primarily a reformulation of ther distress, and were non-judgemental. Importantly, this existing questions) were added as a result of the feedback was from the perspective of the service user. Interviewee provided by those participants who had been interviewed. quotes relating to these issues and the respective items can This increased the item set to 100 potential items for the be found in Table 2. remaining interviews. As a result of the findings from the adult participants, a meeting was held with scientific and Item relevance user group members and decisions were made to remove some items from the set and to change others. This reduced There were some items which were universally liked and had the number of items to 61. Young adults were presented few objections. What these items had in common was that with this reduced item set. Due to the large size of the item the service users could relate to the item as being something set, not all participants provided their views on all items. they experienced regularly. As a result, responses to these The average number of items commented on by the adult questions did not require much thought and were consid- group was 68; the lowest number was 5 by a person who ered easy to answer by the vast majority of participants. was unwell at the time and the interview was terminated Items which fell into this category were ‘I had difficulty early, and also 19 items by a focus group of 7 people. The getting to sleep or staying asleep’; ‘My health limited day maximum number of responses was all 100 items. The aver- to day activities’; ‘I felt able to trust others’ (Quote 1); ‘I felt age number of items commented on by the young adults anxious’ and ‘I felt confident in myself’ (Quote 2). These was 34; the lowest number was 21 and the highest was all items were considered to be particularly relevant because 61 items. The comfort and enthusiasm of the interviewee of the further impact these experiences or feelings had on to continue with the interview was considered at all times. other aspects of their quality of life, for example, if you were The majority of the interviews were recorded and these unable to trust people, then you would never be happy, and recordings were transcribed. Notes were taken for the three how self-confidence impacted on self-worth. adult interviewees who preferred their interview not be Some items were described by the service users as being recorded. All identifying information was removed from irrelevant or meaningless, either to their own mental health the transcripts and notes prior to analysis. problems or to quality of life. For example, when consider- ing the item ‘I felt accepted as who I am’, service users Analysis felt that is was more important to quality of life that they accepted themselves rather than be accepted by others. It A pragmatic approach was taken in the analysis of the inter- was felt that it was not necessary to ‘feel loved’ (Quote 3) view data. From the transcripts, the comments made by each for a good quality of life, and was perhaps a bit of a luxury, participant for each item were charted into a spreadsheet but feeling ‘cared for’ was important and less specific to a framework with items across the horizontal axis and par- certain type of relationship. It was also felt impractical and ticipants on the vertical axis. A traffic light system was used unachievable to have ‘everything under control’ or be able to highlight negative (red), positive (green), and neutral or to ‘do all the things I wanted to do’ (Quote 4). ambiguous comments (orange). Those items with relatively There were objections to some items because they were high levels of acceptability and unacceptability were identi- thought to be too general and not specific to mental health fied. A thematic analysis of the comments was undertaken to difficulties. Examples included ‘I avoided things I needed establish the underlying reasons for the popularity, or lack to do’—it was felt that there may be very good reasons to of popularity, of the items. This information was used as a avoid doing things you did not want to do and doing so could starting point for discussion with the scientific, advisory, and enhance your mental health; ‘I felt irritated’ which was felt expert user groups to establish whether or not an item should to be a normal reaction and not necessarily linked to mental remain as a potential item in the ReQoL measure. In the final health; and ‘I felt tired and worn out’ (Quote 5) which could stages of the development of the measure, this information be due to physical health as well as mental health. A few of 1 3 Quality of Life Research (2018) 27:1893–1902 1897 the young adults thought that being ‘confused about who I Distressing or sensitive items am’ (Quote 6) was a natural thing for individuals in their age group. One of the most common reasons given for objecting to an item, and having the view that it should not be included in a quality of life measure, was that it would cause upset. Some Ease of response items were considered to be too negative. These were often related to suicidal thought and intent. It was the extreme Some items were considered difficult to answer by some negative (and direct) wording within items that participants participants because they were too abstract, thus requiring found distressing, such as ‘I had thoughts of killing myself’ too much thought. Whilst under normal circumstances this (Quote 14) and ‘I thought I would be better off dead’. The would not be a problem, it was something they felt could wording was described as being ‘upsetting’, ‘harsh’, and be particularly difficult when in a distressed state upon first ‘too strong’ and could actually provoke suicidal thoughts. accessing mental health services. This particularly applied to However, the majority of participants thought that sui- items where they thought they were required to consider the cidal intent was an important indicator of quality of life but thoughts of others, e.g. ‘I thought people did not understand expressed a preference for items with a more indirect, sensi- me’; ‘I felt discriminated against’ (Quote 7); ‘I felt accepted tive approach. The items ‘I did not care about my own life’ as who I am’; and ‘I thought people did not want to know me’ and ‘I thought my life was not worth living’ were preferred. (Quote 8). Rather than considering whether they thought peo- Other items considered to be too extreme by some partici- ple understood them (the intention of the item), some would pants, and thus described as upsetting, related to feelings try to think about eg whether people did understand them. about the self. The items ‘I felt humiliated or shamed by There were some items that the service users thought other people’; ‘I felt useless’; ‘I felt shame’; ‘I felt stupid’; may be difficult to provide an honest answer to because and ‘I detested myself’ (Quote 15) were described as ‘too of their own low level of self-awareness whilst ill. It was personal’, ‘embarrassing’, ‘not nice’, and ‘traumatising’. One sometimes only in retrospect that they may realise they participant stated that such a negative question about the self had ‘neglected themselves’ (Quote 9) or were not ‘thinking would make ‘the voices’ more prominent. For items relat- clearly’ (Quote 10). ing to the self, the positive items (e.g. ‘I felt confident in myself’; ‘I felt ok about myself’) were preferential. Of the Item ambiguity negative items, again a gentler approach was favoured, e.g. ‘I disliked myself’ rather than ‘I detested myself’. Some items could be interpreted in more than one way, for Due to its sensitive nature, a number of people responded example, whether an item related to physical or emotional that they would not like to admit to certain things and there- health. ‘I was in pain’ was considered by some to be about fore would either not respond to the item or would answer emotional rather than physical pain, and there was some dishonestly. The reasons given were that they would find it ambiguity whether ‘I had problems with self-care, washing embarrassing [‘I felt humiliated or shamed by other peo- or dressing’ was related to emotional or physical problems, ple’; ‘I had problems with self-care, washing or dressing’ or both. There were also items that were intended to be nega- (Quote 16)] and had concerns surrounding the consequences tive (i.e. indicative of a poor quality of life) but in some cir- of disclosure [‘I had thoughts about killing myself’; ‘I have cumstances could be regarded as positive. For the item ‘I felt threatened or intimidated another person’ (Quote 17)]. guilty’ (Quote 11), a couple of the interviewees indicated that Some items were felt to be insensitive because they were this could be a positive change when they had done something too positive, in particular the items ‘I felt full of life’ (Quote regrettable whilst ill. Similarly, items that were intended to be 18), ‘I felt I could bounce back from my problems’, and, to a positive could be interpreted negatively. For example, ‘think- lesser extent, ‘I felt happy’. Participants felt these items to be ing clearly’ could be due to a total lack of emotion and doing unrealistic, as they thought they were never likely to feel this something ‘bad’ could be ‘enjoyable or rewarding’. For young way even when they were well. Such items were described as adults, the item ‘I felt everything was under control’ (Quote ‘patronising’ and ‘daft’, particularly if asked when they were 12) was not necessarily indicative of good quality of life as it accessing a mental health service for the first time when they meant being in charge, and the item ‘I was able to cope with would be feeling particularly distressed. everyday life’ was preferred as it indicated they were dealing with it without having to take over control. Young people also Judgemental items thought the item ‘I felt hopeless’ (Quote 13) had two interpre- tations of ‘lack of hope’ and ‘feeling useless’. Some positively worded items were thought to be too judge- mental and reflected an opposing value system. This particu- larly applied to those items that were related to doing ‘good’ 1 3 1898 Quality of Life Research (2018) 27:1893–1902 Table 2 Interview participant quotes about potential items for inclusion in the ReQoL measure Quote no. Item Quote (participant) [decision] Item relevance 1 I felt able to trust others “A really good question—if you’re feeling not so good or a little bit paranoid or whatever then you know the trust defi- nitely goes down. And I just think that’s important as well because I think if you don’t feel like you can trust other people then you’re certainly not going to feel very happy” (AI24). [Retained] 2 I felt confident in myself “You need confidence to be able to value yourself—when you have got no confidence and you are down you don’t value nothing” (AF3) [Retained] 3 I felt loved “I think to feel loved might be a luxury—but to feel cared for erm might be sufficient. I mean I don’t suppose the health service can erm prescribe or give love but they can provide care” (AI6) Omitted—item ‘I thought people cared about me’ [Retained] 4 I could do all the things I wanted to do “I don’t think (this question) is very valid—I can’t do all the things I want to do—swim with dolphins, it just isn’t going to happen (laughs). No” (AI12) [Changed to ‘I could do the things I wanted to do’] 5 I felt tired and worn out “… feeling tired and worn out, can be like fatigue or lots of different things […] a lot of teenagers feel like tired and worn out all the time because it’s just kind of how we are, sort of thing, but like if like everything is an effort, like even like brushing your hair is an effort, that’s sort of like different to feeling a bit tired (YPI14) [Omitted] 6 I feel confused about who I am “I think that you’re going to get that with a lot of people really and I think it just like develops, like as you get older you tend to get less-, […] because at that time you’ve got lots of like you’ve got a lot of hormone changes and imbal- ances in your body and it’s just you get mood swings all the time and it’s just like you’re going through lots of different things and it can just be…” (YPI3) [Omitted] Item ease of response 7 I felt discriminated against “I mean I would think if I was answering this and I would look back I would think ‘I don’t know’, I think it’s a hard thing to say that someone has discriminated against you.” (AF5) [Omitted] 8 I thought people did not want to know me “I don’t know what other people think. ‘Didn’t want to know me’, what do you mean by that? … It’s very vague isn’t it, sort of verging on a paranoid thought isn’t it.” (AI34) [Omitted] 9 I neglected myself “I think that when I have been neglecting myself I wouldn’t have known so like maybe I hadn’t showered for like three weeks but I probably wouldn’t realise that I was neglecting myself.” (I31) [Omitted] 10 I was thinking clearly “At the time you think you are thinking clearly, especially if you are in an episode of psychosis what is in your head is very real so I think people could probably answer yes to that one.” [Omitted] Ambiguous items 11 I felt guilty “There are two types there is a good one and bad one, it is good to have guilt because it shows that you are a decent person … it is a good thing, it is another thing that shows you are getting better cos when I was committing my crimes, I didn’t think that I had done owt wrong because I weren’t well like, and when I got well the guilt kicked in, so I think a guilty sign is when you are getting well—but to have too much it can ruin your life can’t it?” (AF3) [Omit- ted] 1 3 Quality of Life Research (2018) 27:1893–1902 1899 Table 2 (continued) Quote no. Item Quote (participant) [decision] 12 I felt everything was under control “Coping is getting through it, just getting through it, but con- trolling is being in charge […] the control may be too much for some people. Do you know what I mean?” (YPI2) [Changed to ‘I felt in control of my life’] 13 I felt hopeless “Erm I think that you could say I felt hopeless at erm at like carrying out a task or…”(I: so for you hopeless there means useless?) “Yes”. (I: you would understand that to mean, I felt useless?) “Yes” (I: Not I felt without hope?) “No I don’t think so [...] I would never think about it like that.” (YPI1) [Omitted] Distressing or sensitive items 14 I had thoughts about killing myself “I don’t like ‘killing myself’ I don’t like that expression, I will be honest with you, maybe ‘have you had thoughts about self-harm’, could it be worded another way? I would agree it is important to ask because lots and lots of people do have suicidal thoughts.” (AI15) [Omitted—item ‘: I thought my life was not worth living’ retained] 15 I detested myself “I think I would be knocked back by it (this question) I think, I would be like ‘oh do I’, you know and then it’s not, it brings on feelings like ‘well yes I am this and I am that’ … I think it might be too strong a word because it might bring back awkward feelings.” (AI26) “’Detest—it might be a bit embarrassing to say and dislike is a bit of an easier one to swallow.” (AI4) [Omitted] 16 I had problems with self-care, washing or dressing “I wouldn’t like to answer that, it’s making me feel a bit ashamed that I can’t take care of myself.” (AI36) [Omitted] 17 I have threatened or intimidated another person “That’s quite extreme isn’t it so I wouldn’t like to answer that question.” (AI38) [Omitted] 18 I felt full of life “I think it’s that thing where people might think ‘well I wouldn’t be here if I was full of life and happy’ .... I think it’s got to have an air of realism about it. For the person who’s reading it to think that it’s a good reflection of how they’re feeling or you know you’ve understood their situation. I just think that they’re a bit too wahooo, here’s your party banner and your balloon kind of thing.” (AI24). [Omitted] Judgemental items 19 I felt I made a contribution/I was able to do things “Is that a sign of being well, you could just be a selfish per - that helped others son? For some people it wouldn’t be part of their life to do something to help other people so there is no relevance. My charity work helps me and helps others—but that is more to do with my work ethic than wanting to help others. If I saw something like that in a questionnaire when I was feeling very poorly it would just make me feel worse because to me that looks like a judgement. You could be making a contribution but it’s actually making you more ill so is it a positive thing or is it actually a negative thing?” (AF1) [Omitted] 20 I lived as independently as I would like to “I’m not sure. I’m not sure if living independently is an imitation of erm good mental health … and I think there’s an implication with being independent that you’re doing alright and if you’re not, you’re not.” (AI6) [Omitted] things ‘I was able to do things that helped others’ (Quote quality of life and that people with mental health problems 19); ‘I felt I made a contribution’ (Quote 19); ‘I did things were not necessarily in a position to be able to help others or that I found worthwhile’; and ‘I felt useful’. It was felt by to do things that were worthwhile, and doing such activities some participants that helping others was not necessary for could make them feel worse. Furthermore, participants noted 1 3 1900 Quality of Life Research (2018) 27:1893–1902 that if the items were answered truthfully (that they did not that a valid quality of life measure should cover the full do things that helped others) this may result in feelings of spectrum of mental health from the worst to the best achiev- guilt. The concept of ‘independence’ was also thought to be able . Additionally the polar aspects of the stages of judgemental because of the assumption that independence recovery should also be represented; from a profound sense was something to be valued (Quote 20). of loss and hopelessness to living a full and meaningful life . So, whilst service user concerns were acknowledged, a decision was made to retain to the next stage of develop- Discussion ment those items which were considered the least extreme and least objectionable to service users but still reflected a Recent guidance advocates clear and transparent reporting of sense of hopelessness, and positive well-being. A similar measure development and assessment of instrument proper- dilemma occurred with items relating to a negative sense of ties . Reliability and psychometric properties of instru- self. The theory of stages of recovery starts with a notion of ments are frequently reported, but a key stage of the develop- ‘negative identity’ through to a ‘positive sense of self’ . ment of any measure is that of content and face validity. The There were no objections related to items having an overly assessment of content and face validity and the acceptability positive sense of self but there were some that caused upset of items by those people who will ultimately use the measure because they were unduly negative (e.g. I detested myself, I can only be achieved through qualitative work with service felt stupid). Again, in order to cover a broad spectrum of ill users . This paper demonstrates the importance of consid- health, those that were negative but considered less insensi- ering service users’ views on potential items. Service users tive were retained. provided their views on potential items to be considered in Some participants objected to items they thought could the ReQoL measure. In summary, they expressed concerns easily be applied to any member of the general public rather about items that were sensitive and could potentially cause than being specific to those with mental health problems distress; judgemental of what was a good outcome; not rel- (feeling irritated, avoiding things, tiredness). After discus- evant to their mental health and quality of life; ambiguous sion, it was decided that this did not necessarily warrant in their interpretation; and difficult to answer either because omission of these items as the measure is designed for peo- they were too vague or abstract. The service users indicated ple at different stages of their recovery. that the potential consequences of including such items To maximise the acceptability, validity, and reliability, would be that they would not respond to the item accurately, the findings from this research informed the development of truthfully, or at all, which in turn would have a detrimental the ReQoL measure at every stage. As a result of the feed- effect on the validity of the measure. This is especially con- back from service users, some items were omitted, whilst cerning as most of the items tested were from, or adapted others were reworded. After psychometric assessment of from, measures currently in use. item performance, the findings were again used to inform Our findings are consistent with those of Crawford et al. the final item selection for the ReQoL measure alongside  who retrospectively sought the views of service users clinician input . about the appropriateness of commonly used measures in For an instrument to measure the concepts most signifi- mental health. One of the primary concerns expressed in cant and relevant to a service user’s difficulties, it is impor - their study was where an item was judgemental in its criteria tant that those providing feedback are representative of the of ‘good’ outcome: for example, the view that people who target population . This was considered as far as possible got on well with family members had better social function- and people with a wide variety of diagnoses, with different ing or quality of life. The items susceptible to this criticism severity levels and from inpatient, outpatient, and recovery in our set of potential items related to the concepts of ‘inde- services were recruited and interviewed. However, it should pendence’, ‘helping others’, and ‘contributing to society’ by be noted that a small proportion of those with the greater doing things that were useful or worthwhile. severity of problems at the time of the interview were less Crawford et al.  also recommended that a measure able, or motivated, to provide a comprehensive response. should not comprise a long list of questions about difficul- They were happy to indicate whether or not they liked an ties associated with mental ill health but rather it should item but did not always articulate why. A greater depth of consist of positive as well as negative items. We did find that response was given by those who had milder conditions or some items could be considered too extreme in their nega- those who were relatively well at the time of the interview, tive or positive nature. Those considered too negative by our though they were able to reflect back on when they were less participants predominantly related to suicidal thoughts and well. Due to time constraints, we were unable to interview those considered to be too positive related to higher levels people from primary care services. of well-being (happiness, joy, and fun). This presented some Whilst care was taken at the initial selection of items concerns for the ReQoL team as it was considered important using an established list of criteria for choosing the best 1 3 Quality of Life Research (2018) 27:1893–1902 1901 items, this stage made it clear that the responses of the ser- References vice users were not always those expected by the research- 1. Department of Health (2010). The NHS outcomes framework ers. This further reinforces the fact that a key component of 2011–2012, London: Department of Health. measure development is that of content validity to the target 2. US Department of Health and Human Services FDA Center population. Here we have shown that certain questions can for Drug Evaluation and Research, US Department of Health be considered irrelevant, judgemental, distressing, ambigu- and Human Services FDA Center for Biologics Evaluation and Research, US Department of Health and Human Services FDA ous, or difficult to answer and care should be taken when Center for Devices and Radiological Health. (2006). Guidance developing a measure to be cognisant of this. However, the for industry: Patient-reported outcome measures: Use in medical items that one population of service users find objection- product development to support labelling claims: Draft guidance. able is likely to differ from another. This paper demonstrates Health and Quality of Life Outcomes, 4, 1–20. 3. Fitzpatrick, R., Davey, C., Buxton, M., & Jones, D. (1998). Evalu- that the involvement of service users within this important ating patient based outcome measures for use in clinical trials. methodological stage is paramount to the development of a Health Technology Assessment, 2, 1–74. reliable and valid measure that will be acceptable to those 4. Streiner, D. L., Norman, G. R., & Cairney, J. (2014). Health meas- who complete it. Further independent research on service urement scales: A practical guide to their development and use, (5th edn.). Oxford: Oxford University Press. users’ acceptance of the measure and items compared with 5. Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Strat- other quality of life measures would be welcomed. ford, P. W., Knol, D. L., et al. (2010). The COSMIN checklist for assessing the methodological quality of studies on measurement Acknowledgements The authors would like to thank all the par- properties of health status measurement instruments: An interna- ticipants in this research and the staff who have been involved in tional Delphi study. Quality of Life Research, 19, 539–549. the recruitment of participants. Special thanks to Rob Hanlon, Jo 6. Patrick, D. L., Burke, L. B., Gwaltney, C. J., Leidy, N. K., Martin, Hemmingfield, and John Kay for providing us with service users’ M. L., Molsen, E., & Ring, L. (2011). Content validity—estab- perspectives. lishing and reporting the evidence in newly developed patient- reported outcomes (PRO) instruments for medical product evalu- Funding The study was undertaken by the Policy Research Unit in ation: ISPOR PRO Good Research Practices Task Force report: Economic Evaluation of Health and Care interventions (EEPRU) Part 2—assessing respondent understanding. Value in Health, funded by the Department of Health Policy Research Programme. 14(8), 978–988. The research was also part-funded by the National Institute for Health 7. Holden, R. R. (2010) Face validity. Corsini Encyclopedia of Psy- Research Collaboration for Leadership in Applied Health Research chology. https ://doi.org/10.1002/97804 70479 216.corps y0341 and Care Yorkshire and Humber (NIHR CLAHRC YH). http://www. 8. Staniszewska, S., Haywood, K. L., Brett, J., & Tutton, L. (2012). clahr c-yh.nihr.ac.uk. The views and opinions expressed are those of Patient and public involvement in patient-reported outcome meas- the authors, and not necessarily those of the NHS, the NIHR, or the ures. The Patient-Patient-Centered Outcomes Research, 5(2), Department of Health. 79–87. 9. Wiering, B., Boer, D., & Delnoij, D. (2017). Patient involvement in the development of patient-reported outcome measures: A Availability of data and material The data that support the findings scoping review. Health Expectations, 20(1), 11–23. of this study are available from the corresponding author. However, 10. Perkins, R. (2001). What constitutes success? British Journal of restrictions apply to the availability of these transcripts as the partici- Psychiatry, 179(1), 9–10. pants only consented to their data being accessible to the researchers 11. Rose, D., Evans, J., Sweeney, A., & Wykes, T. (2011). A model involved in the project. for developing outcome measures from the perspectives of mental health service users. International Review of Psychiatry, 23(1), Compliance with ethical standards 41–46. 12. Groene, O. (2012). Patient and public involvement in developing Conflict of interest All authors declare that they have no conflict of patient-reported outcome measures. Patient, 5(2), 75–77. interest. 13. Ennis, L., & Wykes, T. (2013). Impact of patient involvement in mental health research: Longitudinal study. British Journal of Ethical approval Ethical approval was obtained from the Edgbaston Psychiatry, 203(5), 381–386. National Research Ethics Service Committee, West Midlands (14/ 14. Jacobson, N., & Curtis, L. (2000). Recovery as policy in mental WM/1062). Governance permission was obtained from each of the health services: Strategies emerging from the states. Psychiatric participating NHS Trusts. Rehabilitation Journal, 23(4), 333. 15. Bracken, P., Thomas, P., Timimi, S., Asen, E., Behr, G., Beuster, Informed consent Informed consent was obtained from all participants C., et al. (2012). Psychiatry beyond the current paradigm. British in the study. Participants have consented to the possibility of using Journal of Psychiatry, 201(6), 430–434. quotations in the research reports or other publications, after they have 16. Bohanske, R. T., & Franczak, M. (2010). Transforming public been anonymised. behavioral health care: A case example of consumer-directed services, recovery, and the common factors. In B. L. Duncan, S. D. Miller, B. E. Wampold & M. A. Hubble (Eds.), The heart Open Access This article is distributed under the terms of the Crea- and soul of change: Delivering what works in therapy. American tive Commons Attribution 4.0 International License (http://creat iveco Psychological Association, Worcester, pp. 299–322. mmons.or g/licenses/b y/4.0/), which permits unrestricted use, distribu- 17. Australian Health Ministers Advisory Council (2013). A national tion, and reproduction in any medium, provided you give appropriate framework for recovery-oriented mental health services: Guide credit to the original author(s) and the source, provide a link to the for practitioners and providers. Commonwealth of Australia, Creative Commons license, and indicate if changes were made. 1 3 1902 Quality of Life Research (2018) 27:1893–1902 Canberra. http://www.health.go v.au/interne t/main/publishing .nsf/ 25. Brazier, J., Connell, J., Papaioannou, D., Mukuria, C., Mulhern, conte nt/67D17 06551 4CF8E 8CA25 7C1D0 0017A 90/$File/recov B., Peasgood, T., et al. (2014). A systematic review, psychometric gde.pdf. Accessed 22 Nov 2016. analysis and qualitative assessment of generic preference-based 18. Piat, M., Sabetti, J., & Bloom, D. (2010). The transformation of measures of health in mental health populations and the estima- mental health services to a recovery-orientated system of care: tion of mapping functions from widely used specific measures. Canadian decision maker perspectives. International Journal of Health Technology Assessment, 18, 34. Social Psychiatry, 56(2), 168–177. 26. Connell, J., O’Cathain, A., & Brazier, J. (2014). Measuring quality 19. Boardman, J., & Shepherd, G. (2009). Implementing recovery: of life in mental health: Are we asking the right questions? Social A new framework for organisational change. London: Sainsbury Science and Medicine, 120, 12–20. Centre for Mental Health. https://socia lw elfar e.bl.uk/subject-ar eas 27. Leamy, M., Bird, V., Le Boutillier, C., Williams, J., & Slade, M. /services-clien t-g roups/adult s-ment al-healt h/centr ef orment alhealt (2011). Conceptual framework for personal recovery in mental h/12856 6impl ement ing_recov ery_paper .pdf. Accessed 15 Nov health: Systematic review and narrative synthesis. British Journal 2016. of Psychiatry, 199, 445–452. 20. Mental Health Commission (2007). The journey of recovery for 28. Keetharuth, A. D., Brazier, J. E., Connell, J., Bjorner, J. B., Carl- the New Zealand mental health sector. Wellington: Mental Health ton, J., Buck, T., E., et al (2018). Recovering Quality of Life Commission https://www .mentalheal t h.org.nz/assets/R esour ceFi (ReQoL): A new generic self-reported outcome measure for use nder/te-haere renga -1996-2006.pdf. Accessed 14 Nov 2016. with people experiencing mental health difficulties. British Jour - 21. Hogan, M. F. (2003). The President’s new freedom commission: nal of Psychiatry, 212, 42–49. Recommendations to transform mental healthcare in America. 29. Crawford, M. J., Robotham, D., Thana, L., Patterson, S., Weaver, Psychiatric Services, 54(11), 1467–1474. T., Barber, R., et al. (2011). Selecting outcome measures in mental 22. Boardman, J., & Shepherd, G. (2013) Assessing recovery: Seeking health: The views of service users. Journal of Mental Health, 20, agreement about the key domains. Report for the Department of 336–346. Health. Unpublished (available from the authors). 30. Huppert, F. A., & Whittington, J. E. (2003). Evidence for the 23. Shepherd, G., Boardman, J., Rinaldi, M., & Roberts, G. (2014). independence of positive and negative well-being: Implications Supporting recovery in mental health services: Quality and out- for quality of life assessment. British Journal of Health Psychol- comes. Centre for Mental Health and Mental Health Network, ogy, 8(1), 107–122. NHS Confederation, 34. https ://www.centr eform ental healt h.org. 31. Andresen, R., Oades, L., & Caputi, P. (2003). The experience of uk/recov ery-quali ty-and-outco mes. Accessed 18 Nov 2016. recovery from schizophrenia: Towards an empirically validated 24. Connell, J., Brazier, J., O’Cathain, A., Lloyd-Jones, M., & Paisley, stage model. Australia & New Zealand Journal of Psychiatry, S. (2012). Quality of Life of People with Mental Health Problems: 37(5), 586–594. A synthesis of qualitative research. Health and Quality of Life Outcomes, 10(1), 138. 1 3
Quality of Life Research – Springer Journals
Published: Apr 19, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera