A World Cup penalty shoot-out prompted a breakthrough in the care of patients in Lancashire who have lost the ability to speak.
Intensive care specialist Dr Shondipon Laha was doing his ward round at Lancashire Teaching Hospitals NHS Foundation Trust during the World Cup in 2018 when a patient started mouthing something at him. The patient had undergone a tracheostomy – an incision in the windpipe to help him to breathe – which meant he was unable to speak.
It took a few moments before Dr Laha realised what the patient wanted.
“He wanted me to move out of the way of the TV so he could watch England taking penalties,” he said.
Dr Laha began to search online for lipreading software and came across a technology company, Liopa, that had undertaken years of research in this area.
This led to a partnership between the trust and Liopa, supported by the Innovation Agency, the NHS organisation supporting innovation in Lancashire and the North West Coast.
They then developed an app that can accurately lip-read a patient who – usually temporarily – cannot vocalise.
COVID-19 has meant that speech recognition has become a bigger problem because many patients undergo a tracheostomy as part of their treatment for the virus.
The trust is trialling the SRAVI app – Speech Recognition for the Voice-Impaired – a piece of software that uses artificial intelligence to read the lip movements of someone mouthing words into a mobile phone camera. The information is referred to a library of common phrases and the correct phrase is sent back to the phone which then ‘speaks’ it.
The app has been used by around 20 intensive care unit patients with accuracy of greater than 90 per cent in a six-month trial, says Dr Laha, Consultant in Critical Care Medicine and Anaesthesia.
Before SRAVI, patients who could not speak would point at words on cards or use eye-gaze technology to control a computer screen by ‘scrolling’ with their eyes. These applications were difficult for patients who are too unwell to hold pencils or use complicated technologies.
Dr Laha said: “Ideally, you’d have someone permanently on duty to lip-read the patients but SRAVI gives us an excellent option. You can put it on any device because it uses the phone’s camera, and you get a very rapid response that is usually accurate.
“There’s no confidentiality issue because it doesn’t identify the patient. It will recognise up to 30 phrases and we’re refining it all the time to make it more accurate and expand the number of phrases.
“Not being able to speak is very frustrating for the patient, and for staff. We can deal with patients’ needs much more rapidly and the rehabilitation process is much quicker and smoother if we can improve communication. SRAVI is absolutely fundamental to the successful treatment of patients who have been extremely ill.”
The trust is a regional specialist centre for neurological diseases, so the SRAVI app was used in the treatment of patients suffering from, for instance, stroke and motor neurone disease.
Nathan Armstrong, from Oswaldtwistle near Blackburn, has suffered from paralysis of his vocal cords since he was a child. He underwent a tracheostomy at the Royal Preston Hospital which improved his quality of life but presented communications challenges, so he used SRAVI for around three months to communicate with nurses.
“I am very satisfied with how accurately SRAVI returned the phrases I was communicating while I was in ICU. If you have been on a ventilator you will have been asleep for a week or maybe longer – the last thing you want to do is pick up a pen and try to communicate.
“I think SRAVI will be a good way for people like me to communicate.”
Liopa is a spin-out from Queen’s University Belfast and the university’s Centre for Secure Information Technologies (CSIT), and is the only start-up in the world to focus on automated lip-reading.
At the time Dr Laha contacted them they had not considered a health application for the technology. Now Liopa is in contact with other NHS organisations to explore its potential.
Liopa CEO Liam McQuillan said that continuing to use the technology in intensive care units would help improve its AI models, and feedback from patients and staff would make the technology more usable and establish the optimum length and content of the phrase library.
He added that the Innovation Agency had given the company guidance on National Institute for Health and Care Excellence regulations that would have an impact on their product development, and directed them towards funding for AI projects.
Mr McQuillan added: “The Innovation Agency is a great resource for businesses that are new to the NHS and need help in optimising their products and finding the right people to work with.”
Mike Kenny, Associate Commercial Director at the Innovation Agency, said: “Liopa is the kind of agile, innovative company we like to work with. They’ve come up with an imaginative solution to a pressing challenge and we’re delighted to have supported them.”