Documentation support system

Recording of mock care action with training manikin 

Can we have meaningful AI-supported documentation? Care workers currently need a substantial amount of time for documentation of care actions. From the participatory design workshops we held with care workers and residents, we found out that care workers see potential for supporting documentation processes with technology. We developed a prototype that uses automatic transcription and Large Language Models to extract relevant information from speech during care practices. The aim is to facilitate documentation of care actions by providing summaries of care actions.  

We first recorded fictitious data to test the feasibility of this idea. We recorded the speech of care workers while they were performing care actions with a training manikin in several different scenarios (e.g., dressing, eating and drinking, administering medication). Within this setup, another care worker communicated remotely from another room through the manikin, simulating diverse caregiving situations. The data was used to do first tests with our prototype system. We are doing a series of studies in which we collect audio data of real care practices and develop the prototype further. 

Our first results are described in two workshops papers that were presented at the HRI conference: 

Hirschmanner, M., Grabler, R., Frijns, H.A., Mayer-Haas, E., Vincze, M. (2024) Prototype of a Care Documentation Support System Using Audio Recordings of Care Actions and Large Language Models, Workshop on Human — Large Language Model Interaction HRI ’24, March 11–15, 2024, https://human-llm-interaction.github.io/workshop/hri24/papers/hllmi24_paper_7.pdf 

Grabler, R., Hirschmanner, M., Frijns, H. A., & Köszegi, S. T. (2024, March 11). Privacy Agents: Utilizing Large Language Models to Safeguard Contextual Integrity in Elderly Care [Conference Presentation]. Privacy-Aware Robotics Workshop, HRI 2024, United States of America (the). https://doi.org/10.34726/5960