Research studies often feel a bit like relationships. Some are brief, some more enduring. Often they start out full of excitement, and then as that calms down there is a sense of disappointment: the study did not turn out to be the study you had hoped it would be (many PhDs are like this; mine certainly was). Yet now, for the first time, I have found a study design where it is not just the first excitement of an interesting project. With an amazing research team I am carrying out a randomized controlled trial (RCT) looking at the impact of training in motivational interviewing on the success of workers in engaging parents in child protection work. And this study, for the first time, is true love. This is a type of research I would be happy to do for the rest of my career. Here I want to explain why that is, and argue that more social work academics should fall in love with RCTs.
Social work in the UK has not found RCTs very lovable up to now. Most of the articles written about RCTs in social work are critical. To summarise a complex and nuanced debate, they argue that RCTs are too like science and that the real, social world is too complex. An even clearer indication of our lack of love is that we simply do not do RCTs. There are some in adult services, largely influenced by interfaces with health professions. In children’s services there are a handful: brave pioneering efforts, often looking at specialist services such as different types of fostering. I have looked hard and cannot find a single RCT that looks at how we might do the core business of so many social workers, namely local authority child protection and family support. RCTs are a neglected, misunderstood and unloved study design in social work.
Here I do not want to focus on abstract arguments about methodology or philosophy. Instead I want to write about our RCT and bring alive the sort of contribution that RCTs can make to developing our knowledge base in social work. I hope that by doing so I will demystify RCTs, and perhaps nudge one or two people to try one for themselves: because my guess is that if you try RCTs, you will learn to love them.
Our Study
Let me start by outlining briefly the key features of our study, and then I can highlight what is making it such an enjoyable experience. We are looking at whether training and supervision in Motivational Interviewing (MI) helps workers engage parents, and therefore protect children. MI is a communication method developed in the alcohol misuse field and now tried out with a wide range of behaviour change issues – from drug misuse to diet. MI recognises that it is up to an individual whether they want to make a change. MI skills involve working to engage people in a helping relationship, eliciting from them their motivations for change and agreeing a plan of work. It is fundamentally respectful, collaborative and empowering. In fact, I often describe it as “anti-discriminatory practice in action”.
The first stage of the study was to randomly choose half the workers in one local authority. This group – the MI group – received a 10 week programme of training and consultation in MI last year. The control group of workers will receive the training this year. This piece is not about the findings of the study but I can say that using skills measured in interviews with actors we found a very significant increase in the MI skills of social workers after the training. (Phew!)
The next stage is the heart of the study. For seven months (December to June) all families allocated a social worker in Islington are being randomly allocated to the MI or non-MI group. That is, each family is allocated a worker who was been trained in MI or one who was not. Managers can choose the most appropriate workers from within each group – but they cannot choose a worker from the other group. About 400 families have already been randomized, and the total in the end will be around 500. Some are excluded from the study (we have pre-specified criteria, such as cases where there is an interpreter, families previously allocated to a social worker where reallocation would be appropriate, student social workers and so on). But the main thing is, working in partnership with an amazing local authority in Islington we have randomized in children’s services. Whatever else we find, this means it is possible to evaluate innovations in children’s services using RCTs.
For all families not excluded (about 400) we gather information from workers and the computer system on the progress of families over the first 12 weeks of work. Most importantly, where parents agree we observe and tape an early meeting between worker and parents. We do a structured research interview with the parents shortly after this and then 10 weeks later. This involves standardised instruments to look at parental engagement with the worker and the service, the goals the parents identify for the work and whether they achieve them (using Goal Attainment Scaling) and instruments looking at child well-being and safety, parental stress and anxiety, family functioning and the involvement of other agencies in working with the family. We also look at service outcomes, like whether children come into care, child protection plans and when cases are closed.
A key feature of the study is that there is an expectation that every family allocated is asked whether they wish to participate: it is not up to social workers to make this decision, it is up to parents (with very few exceptions). Here the study has had remarkable success. Around 90% of families have been asked if it is OK for a researcher to observe and tape a session, 75% so far have said yes and most of these parents have agreed to a research interview. These numbers show that when families are asked they are happy to have sessions taped and do research interviews. We currently have about 100 tapes of practice and hope to have twice that.
For us this is an important development. It shows that it is possible to get evidence about direct practice which can then be linked to outcomes for children and parents. This will provide a rich source of evidence not just about the relationship between MI skills and outcomes, but will allow more general analysis of what makes a difference in the way that social workers talk to parents.
I hope that this has given a flavour of the study. Having described the study I want to explain why I am finding it such a positive experience. In doing so I hope to reframe the way social workers tend to see RCTs. For some reason – perhaps because they involve numbers, or come from “science” – RCTs are often perceived as distant from practice. The very opposite is the truth.
What do I love about doing an RCT?
The single most exciting thing about this study is that it is not describing social work or social problems: it is trying to DO something about them. I have done many studies describing a particular problem or issue. In social work research this is what we do more than anything else. But as Marx said, the point is not simply to understand the world but to change it. No slogan should be more important for social work research as an applied discipline. Yet so little of our research actually involves researchers outlining a vision for better practice or policy, working with an organisation to create those changes and then evaluating them. We usually just describe; we sometimes evaluate other people’s projects. Yet the real innovations in helping people, such as MI, have involved researchers and practitioners coming together to develop and evaluate effective ways of working. RCTs are about researchers getting involved in creating and evaluating change.
The second joy of the project flows from this: it is being so closely engaged with practice. I delivered the training and consultations myself, and it has been a privilege to see the commitment and skill of social workers as they struggle to make a difference in often challenging family situations. Every academic should have this type of close engagement with the realities of current practice. Doing so makes us better researchers as well as helping develop better practice.
A particular feature of the study is the opportunity to directly observe what happens in practice. A few years ago I noted that there were virtually no studies looking directly at practice. Even so-called “practice near” research rarely directly observes practice: that is not near enough in my opinion. Pioneers like Harry Ferguson and Chris Hall have been making hugely important contributions in this area in recent years. I am a social worker first, and for me listening to social workers doing social work is the most rewarding type of research imaginable. It is also far and away the best way of researching social work. I cannot understand why we have not been doing this constantly.
The third joy is that because randomization is a robust test of outcomes we are going to learn so much more about the realities of how to develop the skills of workers in the real world. What do I mean by this? Let’s put it this way: if we used what workers said about the MI training we would already have concluded that it was a huge success: most of them were extremely positive about it, some said it had changed their practice completely. We could publish qualitative data that would “show” that MI training “worked”.
Yet in an RCT you realise how complicated change is in the real world. Many of the workers in the non-MI group are outstanding despite not having been trained in MI (darn!). Some in the MI group have not made much change for a whole range of reasons. Most in the MI group are struggling with the competing demands of a full caseload, the need to do all the other things a social worker has to do while still remembering about MI skills. The key thing about an RCT is this: because the only difference between the families in the two groups is whether their worker received the MI package then we can reasonably ascribe any differences between the groups as being due to the MI package. In other words, an RCT is not just an attempt to change the world: it is simultaneously an attempt to robustly test whether in fact you have changed the world. (Or at least the little bit of it you are studying).
It is important that we do this because the danger of getting involved in changing the world may lead us to believe too much in the methods we are championing. I have seen this time and again, and it is human nature. Why go to lots of time and effort to change things if you do not have a passion for the change you want to create? And if you have that passion, why test it as you believe the way of working you promote already works?
This takes us to the heart of perhaps the most contentious claim of RCTs. I believe, in common with most people outside social work, that RCTs are the most rigorous test of whether something works. Carried out properly, randomization should ensure the only difference between the groups is due to the intervention or service you are studying. There is no other way or ruling out tricky issues like people sorting out their problems for themselves or many other sources of bias in findings.
This does not mean RCTs are some sort of panacea. Not all changes can or should be studied using RCTs. Like any other research design there are issues in generalising findings from one situation to another. Yet when they are feasible they are the toughest test of whether a change you are proposing makes a difference. I can see no reason why interventions should not be put to this test.
Indeed, I think the current widespread use of ways of working with people that have not been tested in RCTs is unethical. Time and again social work services are delivered, often on a very large scale, without robust tests of whether they make a difference. In my opinion if social work services tended to be for well-off people this would not be allowed. If teachers, lawyers and doctors were our clients they would – quite rightly – ask what evidence we have for the way we are working. It is scandalous that it is the services for the disadvantaged and marginalised that so often are not based on evidence. It is a source of constant disappointment to me that social work has colluded in this oppressive failure to question what we do with the most robust research methods possible.
Conclusion
So it is these three things that make carrying out an RCT so enjoyable: RCTs attempt to make a difference to social work practice, in doing so they bring you really close to the realities of practice and they then robustly test whether in fact you did make the difference you hoped.
Of course carrying out RCTs is complicated. We have been blessed with a local authority partner in Islington with an extraordinary level of commitment to supporting the project. The workers and managers we work with each day have been amazing, showing a commitment to both MI and the RCT that is exceptional. For me they are epitomising what it means to be a “learning organisation”.
Of course there would be important advantages for our profession as a whole if we embrace RCTs. The most important is external credibility. Politicians and policy-makers respect evidence from RCTs. If we had a body of RCTs showing how different interventions in the field of child protection (for instance) made a difference to children’s lives, we would be in a much stronger position to champion the contribution of social work.
A personal bugbear of mine is that at the moment because we do not do RCTs the lists of “what works” are almost entirely developed by psychologists. And services will increasingly be told to use these interventions. I’ve got nothing against psychologists. Some of my best friends are psychologists. But they do tend to individualise everything. Ground-breaking social work researchers like Lynn McDonald and Martin Webber are using RCTs to explore how we can work to develop social capital. For instance, Lynn’s programme (Families and Schools Together) is being evaluated to see not just whether it improves “parenting” but also whether it increases social solidarity and political engagement. Martin is looking at how to improve social capital for people with mental health issues. These show the enormous contribution that a social work engagement with RCTs could make.
Yet ultimately that should not be the reason for doing an RCT. It is a bit of an incidental by-product. Social workers should do RCTs because they want to try to improve practice not just describe it, and because they want to rigorously test whether they have made a difference in a way that excludes other possible explanations. It is the desire to move my personal research journey from describing problems to helping tackle them that drives my use of RCTs. I hope others – I hope YOU – may find similar motivations move you to try RCTs.
Donald Forrester (@DonaldForr) is the Director of the Tilda Goldberg Centre and Professor of Social Work at Bedfordshire University. Join us @SWSCmedia for a live Twitter Debate on Tuesday (7 May 2013) 8:00 PM GMT / 3:00 PM EDT discussing randomised controlled trials in child protection. Hashtag #SWSCmedia Dr. Forrester has his own blog here
