An ambitious partnership between academia and industry has delivered a ‘breakthrough’ in software technology that will improve the ‘quality, reliability and usability’ of numerical data and solve the ‘data deluge’ industries are facing.
The new concept – the intelligent data quality improver (IDQI) – is an artificial intelligence-based algorithmic technology that can interpret and manage vast volumes of numerical data from a wide range of sources.
The project is being spearheaded by The Data Lab – Scotland’s innovation centre for data and artificial intelligence (AI), the Oil & Gas Innovation Centre (OGIC) and the software company HyperDAP Ltd.
The venture, triggered in response to the data issues industries are facing, launched in 2019 and was conducted at the University of Aberdeen which supported the software research and development process for over a year.
According to the team of experts, industries are struggling to handle, process and interpret increasing quantities of data and, as a result, many data-intensive industrial sectors are at risk of failing to extract valuable information from their production and other experimentally gathered datasets.
Data has become one of the most important assets a business can own, but without proper examination and interpretation ‘it has very little value.’
The IDQI theory has since been tested, verified and validated using a very large oil and gas industrial production and monitoring dataset. It boasts three major benefits.
Firstly, it enables the use of numerical data from past research, development and production which is not currently usable because its poor quality would lead to unreliable or ‘meaningless’ results.
Secondly, it achieves massive time and cost savings by reusing existing datasets, as opposed to acquiring new datasets once again.
Finally, it allows the extraction of further knowledge and information previously masked by the high level of noise and uncertainty that is characteristic of low-quality data.
Gillian Docherty, chief executive of The Data Lab, said: “This wonderful project is a great example of how data, without processing and interpretation, can go to waste. This doesn’t just apply to the oil and gas sector and I would encourage industry leaders from any sector to get in touch with The Data Lab to discover how data can enhance your research and development work.
“Congratulations to everyone at HyperDap and the University of Aberdeen for their tenacity in seeing this project come to fruition. I look forward to seeing how IDQI is developed further.”
Case study: ‘The IDQI solution’
HyperDap, an Aberdeen-based software company, invested in IDQI technology to address the issue of numerical data quality, both in the oil and gas industry, and in many other industrial sectors.
The HyperDAP team is a combination of experienced and young professionals sharing ‘substantial expertise’ in AI and full stack development – ‘ideally equipped’ to take the project and its underpinning software technology to the next level.
The innovation and uniqueness of the IDQI is threefold:
- It will measure numerical quality, proposing strategies to improve it if this falls below a given user-defined, context-dependent level.
- It will evaluate the effect of data quality improvement techniques, proposing alternative ones if dataset quality after improvement still falls short of its target.
- It will learn about user decisions in relation to proposed data quality improvement strategies, using this knowledge to guide and improve subsequent strategies.
These innovations will give IDQI algorithm users the possibility of optimising the quality of existing production datasets in context and of assessing the impact of different computational workflows using their results, enabling a ‘much wider’ reuse of data than currently is the case.
The results
IDQI uses software technologies based on AI in the management of massive datasets to overcome the limitations of current systems. An example of this is the lack of automated data quality evaluation and improvement functionalities and the automated suggestion of optimal workflows based on actual data quality.
The platform will address data quality and its improvement during analysis, focusing on the intelligent specification of different workflows to get meaningful results. It will automatically learn user decisions on machine-proposed workflows, using this knowledge to guide the optimal specification of subsequent workflows.
This innovation uses advanced AI techniques to perform the following:
- Improve production data quality and data analysis.
- Give users the possibility of running a number of different workflows in real time.
- Exploit machine suggestions to identify optimal solutions.
It will allow users to directly specify, program, and run their own workflows without depending on third parties and to exploit machine suggestions at their best. Current systems do not currently include automated mechanisms to evaluate and improve data quality, to recommend alternative workflows depending on evaluation results, or to learn user decisions and use them in the subsequent machine-generated proposal of further optimal actions in this context.
There is indirect positive environmental impact in the oil and gas sector that has been used as a testbed. The technology could help various leading-edge applications such as those analysing oil well integrity and performance to improve safety and decrease the likelihood of malfunctions.
Operators will also be able to identify anomalies that generate environmental pollution from the subsurface into sea water.
Many sectors currently produce large numerical datasets daily, ranging from gigabytes to terabytes. The oil and gas sector is just one such producer.
And according to the collaborators, the IDQI project has the potential to ‘revolutionise data interpretation across all sectors, bringing greater insight than ever before.’
The concept was subject to an independent peer review as part of the Oil and Gas Innovation Centre approval process, where industry experts are asked to comment on the innovativeness and applicability of the project.
One reviewer wrote: “Hydrocarbon accounting is an area of business in which small changes in effectiveness of data handling and systems quality can have a disproportionately large impact on financial gain / risk reduction / accounting robustness.
“As such, it is an ideal area for investment and the application of new technology in pursuit of those business gains. This area of business is also characterised by a large dominance of bespoke systems of very different vintages and technologies, and lack of competition in the service sector providing the niche but essential specialist services in this field.
“Thus, bringing in new ideas, new technology and new players in the service sector will be of benefit. Bringing in the latest computer science technology from academia has the potential to disrupt this market which is likely to be in need of some innovation. Good luck with the project. I look forward to hearing of its successes.”