The EGI infrastructure has been established in February 2010, as a long term sustainable evolution of the EGEE projects. At the same time, the EGI Foundation was created as a governing body to support long term sustainability and to coordinate the infrastructure’s activities. Today, EGI federates more than 20 cloud providers and hundreds of data centres, spread across Europe and worldwide.
I had the pleasure to work in the operations team of the EGI Foundation from almost the beginning, where I had a privileged observation point to appreciate the evolution of the production infrastructure of EGI and its operations activities.
From the beginning, EGI has been among the largest infrastructures supporting research in Europe and worldwide.
The usage of the EGI infrastructure has traditionally been CPU-intensive and the number of CPU cores available in the EGI production infrastructure is a good metric to track its size.
One of the first times I looked into the number of cores in EGI was back in in 2011: the total was about 240,000 cores. A few months ago I calculated about 710,000 cores, which means that the CPU capacity of EGI has almost tripled in only six years of existence. This upturn is responding to the increase of CPU demand of new and existing research communities led by the LHC experiments. The CPU tasks accounted by the EGI infrastructure are currently using ~80% of the available capacity.
Another interesting metric is the number of data centres, or sites as the old-fashioned operations people (like me) call them. The total number of data centres has decreased in the past years, going from ~330 to about ~240 today. The backbone of the production infrastructure is definitely solid and it is here to stay.
EGI’s approach to communities and to our members has greatly evolved during the past years, thanks to the work of all my colleagues. The EGI infrastructure is now more service and user-oriented than ever before. The new service catalogue is much richer, better-structured and fulfills the requirements of new research communities. In parallel, the services that are offered to the members of the federation have been consolidated with a plan for a long-term, fee-based support. These services include for example service monitoring, accounting, security coordination and helpdesk.
The operational tools in this portfolio have evolved as well to be more reliable and flexible. For example, monitoring is now centrally provided by ARGO to better support new monitoring probes and be able to react to requirements quickly; the Operations Portal now allows browsing the information system in a GUI, and the messaging network has a new HTTP API that can be easily re-used by new use cases.
Looking at the current service portfolio, it can be noted how the burst of new services involved to some extent the federated cloud. And it has been a great achievement of EGI to design, build and bring to production a new set of services, based on new technologies that brought new capabilities. A lot of hard work has been spent from 2014 onward in tuning the new services to the operational framework of EGI, and the other way around, in adapting the operations to the new requirements. Security coordination evolved the security operations to deal to the “new dimension” added by the cloud, and so did the operational tools and support activities.
What are the future challenges for the EGI Operations? The EOSC-hub project will start in January 2018, and it is the first EOSC-related project that focuses on the provisioning of production services. EGI already has a solid experience in providing production services, but the project brings together both EGI providers and external ones. The challenge will be to integrate the various best practices implemented by the service providers and find a lowest common denominator to build what will be the EOSC operational framework.
The biggest contribution of EGI Operations to the future European Open Science Cloud is our combined experience (EGI Foundation and 38 operations centres) in building production operations.
It has been a pleasure to work together and to see how EGI evolved during these years, and also an honour to be able to give my small contribution to this process. While leaving my position at the EGI Foundation, I would really like to thank everybody who contributed at every level to the EGI Operations during these years, and wish all the best for the exciting times ahead for EGI!