How EGI and DIRAC join forces to develop a new service

Yin Chen and Baptiste Grenier on the achievements of the collaboration

DIRAC was originally developed to support the data management and processing of the LHCb experiment at CERN. Today it acts as a complete solution to support grid, cloud and HPC resources, targeting various large scientific communities including LHCb, Belle II, EGI, CTA, GridPP, WeNMR, VIP, FranceGrilles, SKA.

DIRAC provides complete solutions for production managements, handling distributed large volumes of scientific data and optimising job executions.

The software package includes:

  • A data management system comes with access protocols to various storages, reliable data transfer (based on FTS), file fatalogues allowing metadata-based data-discovery, and optimising routine data distribution tasks.
  • A workload management that allows to aggregate resources of different source & nature in a single system computing, and optimise the job submission.
  • A flexible transformation system that provides automated data-driven submission of processing jobs with workflows of arbitrary complexity
  • An accounting system responses to collect and store data regarding to the activities, and a monitoring system to monitor various components.
  • A webportal with an easy access user interface.

The modular organisation of the DIRAC components allows selecting a subset of the functionality suitable for particular applications or easily adding the missing functionality. These are very useful for communities to have customised environments for handling their own data. DIRAC evolves over time and in recent development, it includes new features such as supports for Cloud, HPC, muti-VO etc.

In order to better support research communities, EGI collaborated with DIRAC to develop a new service: the EGI Workload Manager, also known as DIRAC4EGI. The service is coordinated by the EGI Foundation and operated by IN2P3 on resources provided by CYFRONET.

The EGI Workload Manager is already used in production by early adopters like WeNMR, that were able to easily switch their Science Gateways from gLite WMS to DIRAC. The work is supported by the WeNMR Thematic Service under the EOSC-hub umbrella.

The accessibility of this service will also be extended soon: currently, authentication is based on the old X509-based model that is well known in the distributed computing for research world, but in the context of the EOSC-hub project, this service is being integrated with the EGI Check-in service. This will allow users to seamlessly connect using their institute’s Identity Providers without having them to own a personal grid certificate.

EGI has a formal procedure to include production-level services into the EGI service catalogue, including a complete business plan for service provisioning, assurance of resource capacity, full documentation, service order and help desk. EGI is now in the final stage to sign an OLA with the technology and resource providers of the DIRAC-based Workload Manager. With an OLA in place, this service will be officially included into the EGI service catalogue and made openly accessible to the EGI community.

More information

The 8th DIRAC user workshop was held 22-25 May in Lyon, and hosted at CC-IN2P3 by CNRS.

Yin Chen is Senior Technical Outreach Expert at the EGI Foundation.

Baptiste Grenier is Senior Operations Officer at the EGI Foundation.

Subscribe to the EGI newsletter: