Service Management as a Service helps projects establish a fully functional Service Management System (SMS) for new e-Infrastructures, whether domain-specific or regional
Service
FedLLM
Use Large Language Models securely with full control over your data
Overview
FedLLM
The LLM Service provides secure, high-quality access to Large Language Models through a web interface and an OpenAI-compatible API. Users can generate and analyse text, run retrieval-augmented generation (RAG) workflows, and integrate LLM capabilities into scientific applications.
The service ensures full transparency on data processing and supports authentication and authorisation via the EGI Check-in service. LLM inference can be federated across multiple providers, including commercial infrastructures, while keeping user data safe and private.
Main features
FedLLM main features include:
- Access to Large Language Models (LLMs) through a web-based chat interface
- OpenAI-compatible REST API endpoint for programmatic access and workflow integration
The service is TRL 7-9 depending on the components.
This service is part of the EGI EOSC Node service offering
Services provided through the EGI EOSC Node are offered in accordance with the EGI User Access Policy (UAP) and Acceptable Use Policy and Conditions of Use (AUP).
Service Offer
1
LLMs for Researchers
Users can directly access the service UI or the API with a valid EGI Check-in account.
2
LLMs for Communities
- Community requests service specifying LLMs options, tokens per minute (TPM) limit and any additional extra requirements for the LLMs environment.
- SLA is negotiated with VO, specifying resources to be made available and how to fulfil extra requirements.
- A dedicated instance of the service is deployed for the VO on top of the allocated resources and configured to allow access members of the VO according to the VO policies.
- Users log in and start using LLMs as needed.
Service documentation