How Big Data Test Infrastructure helped
The Big data test infrastructure (BDTI) is an open-source analytics cloud stack that promotes the reuse of public sector data. It helps public administrations and projects led by the public sector improve the experience of citizens, make government more efficient, and boost business and the wider economy through data.
In this case, the project "BDTI (Big Data Test Infrastructure) eProcurement Pilot" involved Italian, Portuguese, and Norwegian authorities and centred around providing a scalable virtual environment and analytics routines to work on procurement data and support the creation of the eProcurement data space.
Thanks to BDTI, instead of setting up a testing environment for these solutions, it is possible to use its pre-prepared infrastructure to allow public administrations to concentrate on the core business and insights, maximising the value obtained from their data.
The BDTI infrastructure was assessed as particularly useful to support public sector partners in the pilots that do not regularly use their own internal tools but with open datasets already available in their data catalogues in different formats (e.g., JSON, CSV).
Specifically, the BDTI e-Procurement Pilot represented a first step in implementing the ongoing EU Public Procurement Data Strategy, which foresees the development of an infrastructure where data from various sources are harmonised and made available for reuse and analysis.
Background
Every year, over 250 000 public authorities in the EU spend around 14% of the EU GDP (about €2 trillion per year) on the purchase of goods, services, and supplies. Public procurement is important, as public authorities are the principal buyers in many sectors such as energy, transport, waste management, social protection, and the provision of health and education services. The public sector can use procurement to boost economic growth and job investments and to create an economy that is more innovative, energy and resource efficient, and socially inclusive. High-quality public services depend on modern, well-managed, and efficient procurement. Improving public procurement can yield big savings; even a 1% efficiency gain could save €20 billion per year. Despite the various efforts to standardize procurement at the national, EU, and international levels and the availability of digital negotiation instruments, the digitalization of public procurement is still facing some challenges.
During 2020, Italy issued a request to the ISA² Programme to support the development of digital solutions that would enable public administration, businesses, and citizens in Europe to benefit from interoperability and cross-border public procurement services. The request was to provide a common framework, using open-source software, and an infrastructure for monitoring public procurement in the EU through a set of analytical services and tools based on a common data model to foster effective government spending, active common policymaking, and competitiveness. This request triggered the setting of a pilot to explore and pave the way to the EU Public Procurement Data Space.
The challenge addressed through BDTI
The challenges facing the digitalisation of public procurement comprise insufficient data sharing and reuse, the inability to match correlated data from different databases, and insufficiency in overall data quality. Another bottleneck relates to the shortage of experts with the skills and knowledge about the specific procurement domain and the required expertise to use digital instruments and perform complex analyses. All these factors prevent the implementation of effective data-driven "public procurement" focusing on efficiency and cost savings, which go hand in hand with digitalisation. There is a clear need for a common European approach to address these challenges many EU countries face.
Data transformation process in the BDTI
To carry out the project, the BDTI team collaborated with pilot participants to design and implement a process to transform the data, starting from datasets located in the cloud storage space of the BDTI, into a knowledge graph (KG) by leveraging the Resource Description Framework (RDF) Mapping Language (RML), using a set of its functions for data manipulation purposes. In the RML mapping rules, links were created to other linked open datasets available in the Web of Data, such as controlled vocabularies published by the Publications Office and recommended in the eProcurement ontology.
Figure T1. Adding a map chart to the dashboard
For what concerns the eProcurement ontology concept, procurement data has been identified as data with a high-reuse potential. Given the increasing importance of data standards for eProcurement, a number of initiatives driven by the public sector, the industry and academia have been kick-started in recent years. The vocabularies and the semantics that they are introducing, the phases of public procurement that they are covering, and the technologies that they are using all differ. These differences hamper data interoperability and thus its reuse by them or by the wider public. This creates the need for a common data standard for publishing procurement data, hence allowing data from different sources to be easily accessed and linked, and consequently reused.
The RML mapping rules were expressed in R2RML syntax, a language for expressing customised mappings from relational databases to RDF datasets. They were saved in the cloud storage space of the BDTI and executed using the RML mapper through instructions configured in a workflow management system.
Figure 2. Pilot architectural scenario
BDTI as a tool for solution
Thanks to the BDTI eProcurement Pilot project, it was possible to have:
- A streamlined data transformation from a wide variety of existing EU and national data schemas to the eProcurement ontology;
- A standardized check on the quality of procurement data quality, supported by a generic data quality dashboard built upon the results of the Testbed services, allowing for the assessment of the quality of the data used in the analysis;
- Data combination from various and heterogeneous data sources;
- A transformation of data for statistical use through sample analytical tools providing time-series and predictive analysis with multiple variables to prove the feasibility of addressing the need for public procurement strategic objectives. In this case, the aim was to foster competition.
Key Takeaways
With digital tools, public procurement becomes more transparent, evidence-oriented, and optimized, fostering competition. Other significant benefits are reusability, sustainability, significant savings, increased transparency, greater innovation, and new business opportunities by improving the access of enterprises to public procurement markets, including SMEs. The e-Procurement framework will also help in the ambition to support strategic objectives throughout public procurement data analysis and foster cross-border interoperability.
Interested in launching your own data pilot? Get in touch with EC-BDTI-PILOTS@ec.europa.eu
References
• Original paper: Semantic Knowledge Graphs for Distributed Data Spaces: The Public Procurement Pilot Experience
• Gitlab repository
• Study on up-take of emerging technologies in public procurement
• SME needs analysis in public procurement report
• Internal Market, Industry, Entrepreneurship and SMEs
• Green paper on expanding the use of e-Procurement in the EU
• Directive of the European Parliament and of the Council on public procurement and repealing Directive
• Report from the Commission to the European Parliament and the Council on the review of the practical application of the European Single Procurement Document (ESPD)