Multimodal Large Language Models for Patient-Generated Data in Shared Decision Making (DC7)

This PhD project is embedded in Work Package 2: Technologies for Multimodal Data of ENDOTRAIN and will advance the development of multimodal Large Language Models (LLMs) for integrating time-series physiological data, clinical assessments, and natural language summaries in the context of endocrine disorders, with a focus on adrenal insufficiency. 
 
The project will: 

• Design and evaluate multimodal AI architectures that combine wearable-derived time-series signals, structured clinical data, and patient-generated language reports.
• Explore advanced fusion strategies for integrating heterogeneous modalities into context-aware clinical decision support systems.
• Create visual and language-based representations of complex patient data to support shared decision making (SDM) between patients, caregivers, and clinicians.
• Conduct participatory research with ~40 patients living with adrenal insufficiency using surveys, interviews, and workshops.
• Produce publishable contributions at the intersection of AI, Human-Computer Interaction, and Digital Health, with targeted venues including AAAI, ACL, NeurIPS, ICML, ACM CHI, and ICHI.
• Contribute to ethical, explainable, and responsible deployment of multimodal LLMs in the management of endocrine diseases. 

Research Fields: 

Artificial Intelligence, Multimodal Machine Learning, Natural Language Processing, Human-Computer Interaction, Digital Health, Endocrinology 

Secondments (Preliminary Plan): 

• UiB (Norway): 1–2 months — Patient and caregiver interviews, exploration of legal/ethical aspects, and integration of multimodal datasets.
• Karolinska Institutet (Sweden): 1 month — Development and evaluation of tailored self-reporting and visualization interfaces for endocrine patients.
• University Hospital Zurich (Switzerland): 1 month — Advanced analysis of patient-reported outcomes and integration with multimodal AI approaches. 

The candidate should work full-time on the project and must not have resided or carried out their main activity (work, studies, etc.) in Spain for more than 12 months in the 36 months immediately before their recruitment date.  

The student will be enrolled in the PhD programme at UiB and participate in the interdisciplinary training of the ENDOTRAIN network, which includes: 

• Advanced scientific and transferable skills courses
• International workshops, summer schools, and retreats
• Cross-sector training and secondments with academic, clinical, and industry partners 

Required: 

• MSc (or equivalent) in Mathematics, Computer Science, Computational Linguistics, Engineering, or Medical Engineering
• Strong expertise in Artificial Intelligence, Machine Learning, and Natural Language Processing
• Experience with multimodal learning (integration of time-series, clinical, and textual data)
• Strong programming skills (Python, PyTorch/TensorFlow)
• Demonstrated coding portfolio: GitHub link required
• Excellent written and spoken English (working language of the project)
• Compliance with MSCA mobility rule: applicants must not have resided or carried out their main activity in Spain for more than 12 months in the last 3 years before recruitment 

Desirable: 

• Experience with LLMs, multimodal fusion, or explainable AI
• Background in digital health or clinical datasets, ideally related to endocrine diseases
• Record of publications or strong aspiration to publish in top-tier AI or HCI venues (AAAI, ACL, NeurIPS, ICML, ACM CHI, ICHI)
• Interest in responsible AI and patient-centered healthcare innovation 

• A 3-year full-time employment contract under Spanish labour regulations (contrato de trabajo).
  • Contract covers tuition fees, training support, research costs, secondments, and travel allowances in line with MSCA Doctoral Network rules.
• Gross salary range: €41,000 gross salary (to be completed based on the host institution’s MSCA budget and Spanish regulations).
  • Salary includes the MSCA Living Allowance, Mobility Allowance, and, where applicable, the Family Allowance (€4,495).
• Doctoral candidates employed in Spain are fully integrated into the national Social Security system (Seguridad Social), which provides:
  • Public health insurance (universal healthcare coverage)
  • Pension contributions to the Spanish state pension system
  • Unemployment insurance
  • Maternity/paternity leave entitlements under Spanish labour law
  • Occupational accident and illness insurance
  • Access to other standard employment rights (paid vacation, sick leave, etc.)
• Training, Mobility & Networking Opportunities
  • Dedicated travel and secondment budget, enabling national and international research stays within the MSCA network.
  • A comprehensive training programme including thematic workshops, transferable-skills schools, and network-wide events.
  • Opportunities for international networking, industry exposure, and collaboration across sectors.
  • Support for career development, including conference participation, mentoring, and collaboration with industrial and academic partners.

Applications will only be accepted through the Jobbnorge portal.

Applications without the mandatory attachments (application, CV, mobility declaration, motivation templates) will not be considered.

Informal inquiries may be directed to: Dr. Luis Fernandez-Luque (Adhera Health SL)