The eXtreme Data Cloud project (XDC) started in November 2017 as a follow-up of the successful INDIGO-DataCloud project and will last until January 2020. XDC develops scalable technologies for federating storage resources and managing data in highly distributed computing environments. The initiative is funded by H2020 under the EINFRA-21-2017 call “Computing e-infrastructure with extreme large datasets”, with the consortium made of eight partners – INFN, University of Cantabria, DESY, CERN, CNRS, AGH, ECRIN ERIC, EGI Foundation – and coordinated by INFN.
XDC brings together technology providers with long-standing experience in software development and research communities belonging to a broad spectrum of data-intensive scientific disciplines, such as life sciences, biodiversity, clinical research, astrophysics, high-energy physics and photon science. The different communities provide concrete use cases and heterogeneous requirements in the field of access and management of data volumes at an unprecedented, “extreme” scale. XDC will integrate some intelligence on top of data management functionalities to meet the specific needs of scientific communities and implement a more flexible and efficient computing infrastructure in Europe. The project’s services are developed through the use of standards and protocols available on state-of-the-art distributed computing ecosystems. They can be easily plugged into European e-Infrastructures and in general on cloud based computing environments such as EGI, the European Open Science Cloud (EOSC) and the Worldwide LHC Computing Grid (WLCG).
XDC will release open source software based on already existing components enriched with new functionalities and plugins that can be adopted by as many user communities as possible. Production quality services such as EOS, dCache, Onedata, the Indigo-PaaS Orchestrator will all be integrated into the XDC service catalogue with improved functionalities concerning policy-driven data management, quality-of-service based data movement, smart caching, metadata management, secure storage and encryption.
Services provided by XDC will be scalable to cope with extreme scale scientific experiments like those run at the Large Hadron Collider at CERN and the Cherenkov Telescope Array. Given the XDC software will be released as Open Source platforms available for general exploitation, we foresee a much wider adoption of XDC functionalities by less experienced and smaller user communities as well.