Published:
Professor David Flynn, Director of the Smart Systems Group at Heriot-Watt University, explores how industry, society and government extract future value from Big Data.
Many businesses, organisations and government departments are increasingly interested in unlocking the value of big data analytics to gain a detailed insight into their stakeholders and market trends.
However, this fascination with big data (BD) and desire to be seen at the forefront of new technology is leading many companies, as well as a number of projects supported by public funds, into wasteful investments and inefficient use of staff and resources.
Part of the problem is that key decision-makers approach the process of discovering and leveraging useful information from data in the same way as oil discoveries; given sufficient investment they will eventually yield the knowledge required to realise the gains.
Yet, using and exploiting data requires not just new technical knowledge and analytic skills, but also proper consideration of privacy and security responsibilities.
Even before starting the process of recovering large-scale data, organisations need a well-defined strategy on how they will derive value from this data and maintain their initial and ongoing stewardship of that data once it’s in their possession.
Any company looking to derive value from BD needs to consider their data governance, ensuring they understand their role and responsibilities and how the relationships with legislative bodies, customers and stakeholders affect the who, what and how big data is accessed. This then shapes their organisational and data management needs, including security, so they can begin optimising on what it is they really need in terms of BD to address well-defined challenges.
In the context of critical infrastructure, autonomous systems or advanced consumer products, meaningful discovery in applied machine learning and data science is about working on well-defined challenges, maximising existing systems and data sources and embedding existing human knowledge into the analysis and verification stages of BD.
Examples include ReFLEX Orkney, the UK’s largest whole system project wherein partners support the creation of solutions from Big Data Analytics (BDA) with the clear aims of supporting decarbonisation, social justice and equity in the energy transition, and resilience in energy services.
Another example is the ORCA Robotics – Robot and Asset Self-Certification, wherein well-defined capability challenges from the industry shape the research and verification of autonomous systems and AI solutions.
Working with domain experts and capturing the experience and domain knowledge they have, we can then co-create useful solutions aligned with our ambitions. It needs to also be accepted, that irrespective of the demand or urgency, this is also not going to be an overnight success story.
Teams that can accelerate discovery and innovation in BDA requires the assembly of the right team – with the right mix of expertise and experience, and supported by the resources and incentives that can deliver sustainable innovation.
Academics are now exploring the benefits of a highly multidisciplinary research environment, to deliver solutions leveraging big data to support the energy transition. A current example of this work is a partnership project with SP Energy Networks.
To set the BD challenge, the number of smart meter readings for a large utility company is expected to rise from 24 million a year to 220 million per day.
By bringing the right team together, from the outset, we were able to co-create solutions using BD that are aligned to ambitious goals, that had the resilience to cope with the imperfect situation the real world commonly presents us with.
As reported in our research:
Automating the verification of the low voltage network cables and topologies
Predicting the voltage distribution for low voltage networks using deep learning
This work has enabled us to accelerate network analysis design times by 67%, backfill imperfect records to improve network asset management and planning, and within a national context that could save network operators £1bn in investment costs, whilst also accelerating access to distributed renewable generation and green (EV) transport for customers.
Our team has also explored issues around inclusion in the energy transition using data-driven analysis, as well as presenting “comprehensive reviews” on the challenges and opportunities in AI and Machine Learning approaches to energy demand response Additionally, we have conducted research into machine learning approaches to energy demand-side response.
As the world pushes the timelines on the pathways to discovery and innovation, it’s important that universities are engaged in this process in order to accelerate the creation of effective and sustainable solutions in applied big data analytics.
Professor David Flynn
Director of the Smart Systems Group at Heriot-Watt University