Big Data Test Infrastructure (BDTI) resources, include technical documentation, deployment guides, reusable code, and best practices, to support public administrations and organisations.
Featured resources
To support public administrations and organisations in making the most of big data and analytics, we have gathered all BDTI resources in a dedicated Gitlab repository. This includes technical documentation, deployment guides, reusable code, and best practices from our pilots and success stories.
An introduction to geospatial analytics
Recognise patterns in data by leveraging geographical, spatial, and location information.
- Learn about Geospatial Information Systems (GIS), their importance, and how they help with modern challenges in public health, climate change, smart mobility, urban planning, and more
- Get to know the two most common Geospatial data types: Vector data and Raster data
- Explore the Geospatial Analytics Extension for KNIME (co-developed by CGA Harvard) which can be used to access, transform, manipulate, and process GIS data in a low-code fashion
Learn the fundamentals of statistics and its necessity for data analysis.
- Delve into descriptive statistics for a selected social phenomenon in the EU, focusing on measures of central tendency, variability, shape distribution, association and exploratory plots.
- Get to know the concept of probability, exploring different event types, probability calculations and more advanced topics like conditional probability and the Bayes’ Theorem.
- Explore the Statistics extensions for KNIME, which can be used to detect outliers, obtain correlation matrices, test hypotheses and more in a low-code fashion.
An introduction to graph analytics
Analyse complex relationships between entities and uncover patterns in data.
- Learn theoretical basic concepts about graphs, including terminology
- Learn about applications of graphs focusing on use cases involving trip planning to, and within, the European Union
- Learn how to model, process, analyze, and visualize graphs without any coding with KNIME Analytics Platform
Dashboards for data visualisation: Analysing and presenting traffic accident insights
Transform complex data into clear, actionable insights, enhancing your ability to communicate effectively.
- Gain a thorough understanding of data visualisation techniques and their applications in urban safety analysis
- Get hands-on experience in managing and visualising static datasets
- Learn skills in data preprocessing, storage, and integration using PostgreSQL
Communicating complex datasets: Integrating real-time data for urban insights
Learn to integrate and visualise real-time data.
- Build a foundational understanding of data visualisation techniques and tools
- Get experience in integrating and visualising real-time data
- See practical applications of data visualisation in public sector decision-making, particularly in urban planning and environmental management
Harnessing climate data: Classification and predictive analytics for tourism
Make accurate predictions and informed decisions by uncovering patterns and trends in historical data.
- Understand data classification and predictive analytics
- Learn to use Jupyter Notebooks for data processing and MinIO to store and manage the data securely
- Develop experience in applying machine learning models to real-world data
Predictive modelling and real-time analysis: Real-time forecasting and monitoring of bicycle use
Analyse historical data to uncover patterns and relationships to forecast future events, identify trends.
- Gain a solid understanding of predictive modelling using real-world data
- Get experience with data tools like R Studio and MongoDB
- Get practical skills into setting up automated data pipelines
Natural language processing & data visualisation: Enhancing citizen participation
Apply data-driven techniques to understand survey results better and uncover actionable insights.
- Learn to use NLP in translating texts and analysing sentences
- Get experience in analysing survey responses in bulk
- Build skills in managing data storage and processing using Apache Superset, MongoDB and Jupyter Notebooks
Showcasing innovation panel discussion: EU public administrations' data-driven use cases
Discover how data-driven insights, powered by open data, can revolutionise public services across Europe, turning complex challenges into manageable solutions.
BDTI Pilot Showcase: Presenting data-driven solutions in action
Presentations from project managers who have leveraged the BDTI to tackle key challenges and deliver impactful results.
A free course to help public administrations explore tools offered on BDTI through a practical use case. The course guides participants through a typical data project workflow following a fictional use case. Course slides and live recordings are available at the links below.
- Session 1: Data Access and Exploration: Lay the foundation for data analysis by loading and exploring the relevant datasets.
- Session 2: Data Cleaning and Transformation: Prepare the data for analysis by cleaning and transforming it.
- Session 3: Data Blending and Storage: Learn techniques for automating data blending and storage.
- Session 4: Basic Analytics: Begin the analytical process by addressing the core objectives.
- Session 5: Advanced Module: Gathering Data from the Web and Geo Visualisation.
Use cases inventory: Inspiration and templates to build your own use case.
These real-world use cases illustrate how data-driven insights, powered by open data, can equip governments with the ability to quickly respond to challenges, improve efficiency, foster transparency and create more effective policies to benefit citizens. Reviewing the use cases can inspire public administrations to design and implement their own data-driven approaches.
Data use case templates
These materials aim to support public administrations in defining and designing data use cases.
BDTI practical resources: This document provides resources such as questionnaires, checklists, decision trees and frameworks.
Project scoping methodology: This document provides guidance for public administrations interested in experimenting with data projects. The objective of this document is to provide useful tools for users to define and design their pilot use case.
BDTI milestones
-
2019
BDTI launches and first pilots begin.
-
March 2019
Hackathon data collection: BDTI deployed a dedicated instance to support a custom smartphone data collection application, aggregating approximately 1.5 TB of highly personalised data over two weeks and securely persisting it in S3 with strict privacy and access controls.
-
May 2019
Online job vacancies: A big data platform for a data lab that allows statistical offices in EU countries to explore and process data collected by RLMI for real-time labour market insights.
Automatic identification system: The pilot aimed to use big data on the geo-positioning of ships generated Automatic Identification Systems (AIS) to enhance the quality and comparability of existing maritime statistics and produce new statistical products.
-
2020
Pilots continue, and the BDTI team engages with public administration and academia across Europe.
-
February 2020
Norway, Difi (agency of public management and eGovernment): Created a data lake using public procurement data from PEPPOL, enabling transactional data analysis to enhance procurement efficiency, detect inefficiencies, and provide insights for further improvement.
-
April 2020
City of Valencia, Spain: A pilot to extract the knowledge contained in a large quantity of existing scientific evidence and regulation documentation on COVID-19, and provide it to the clinicians and managers in a manageable way by means of advanced data visualisation tools.
Municipality of Milan: A predictive modelling framework for analysing citizen mobility data in Milan during Covid-19 Phase 2, enabling informed policy-making through data-driven insights.
-
July 2020
City of Florence: A pilot to understand Covid-19 impact on mobility by leveraging data collected by Smart City Control Room (SCCR) - as a key part and enabler of a bigger project called REPLICATE - to ensure lockdown measures were lifted in a responsible and controlled manner.
European Blood Alliance and European Commission (DG SANTE): A pilot to create and manage an EU-wide open-access platform that collects data to support a study on Covid-19 convalescent plasma therapy.
-
September 2020
Italy, Portugal and Norway: The pilot involved Italian, Portuguese, and Norwegian authorities and centred around providing a scalable virtual environment and analytics routines to work on procurement data and support the creation of the eProcurement data space.
-
March 2021
Municipality of Casola Valsenio: To analyse population distribution and internet accessibility in Casola Valsenio to inform decision-making for future broadband infrastructure deployment and environmental monitoring initiatives.
-
May 2021
Municipality of Casola Miglierina: To analyse energy consumption patterns and optimise renewable energy production using advanced data analytics techniques, including predictive modelling and time-series analysis.
-
2022
Results start to prove how BDTI is helping public administrations and projects led by the public sector improve citizen experience, make government more efficient and boost business and the wider economy through big data.
-
2023
Pilots continue, and knowledge-sharing communication activities begin.
-
March 2023
A dedicated BDTI website is launched, detailing the BDTI service offering, sharing resources and pilot success stories.
-
May 2023
The BDTI team hosts data skills webinars and workshops to encourage public sector data literacy.
-
October 2023
Dun Laoghaire Municipality, Dublin: A pilot to harness integrated traffic and event data for strategic urban planning and provide actionable insights to local authorities and communities to support sustainable urban mobility initiatives.
BDTI team attends the Open and Agile Smart Cities Summit and the European Week of Regions, presenting the solution and hosting use case workshops.
The BDTI Kitchen Newsletter launches, delivering monthly BDTI news, events and public sector data analytics insights.
-
January 2024
GRNET and University of Macedonia, Greece: GRNET and UoM launch a pilot to transform MITOS, which provides structured descriptions of over 3,000 public services, into Linked Open Data.
The BDTI team attends and presents at OASC 2024.
BDTI Essentials online course is launched: "Enabling a Data-informed Public Sector: An Introductory Course to BDTI Essentials”, fostering a data-literate and innovative public sector across the EU.
-
February 2024
The BDTI Kitchen Newsletter launches, delivering monthly BDTI news, events and public sector data analytics insights.
-
March 2024
BDTI promotional materials are translated into all 24 European languages.
-
May 2024
City of Bochum, Germany: A pilot to build a machine learning model to revolutionise how tree health is monitored and predicted in urban areas.
City and University of Turku, Finland: A collaborative pilot to prepare the analysis of traffic flows to improve public transport by combining several mobility data sources with geodata.
-
July 2024
BDTI Skills Studio launches offering workshops and webinars covering various topics of data analytics, allowing participants to gain practical knowledge and hands-on experience.
City of Naples, Italy: A pilot to enhance urban planning and mobility strategies through advanced analytics on public space and mobility-related data.
-
October 2024
The BDTI team host a workshop discussing how regions and cities can benefit by reusing public sector information for innovation at the Smart Country Convention in Berlin.
City of Arezzo, Italy: A pilot to foster data-driven decision-making to achieve a sensitive decrease in the city’s scores of accidents and injured people using an in-depth, overarching analysis of a vast database of dangerous roads and intersections in Arezzo’s municipality.
-
November 2024
The BDTI team host a workshop on the importance of reusing data for the future of cities and regions at the FARI Brussels Conference.
-
January 2025
Pilot participants present their case studies at the BDTI Pilot Showcase virtual event.