Breakthrough in software technology to solve ‘data deluge’ faced by industry
An ambitious partnership between academia and industry has delivered a ‘breakthrough’ in software technology that will improve the ‘quality, reliability and usability’ of numerical data and solve the ‘data deluge’ industries are facing.
The new concept – the intelligent data quality improver (IDQI) – is an artificial intelligence-based algorithmic technology that can interpret and manage vast volumes of numerical data from a wide range of sources.
The project is being spearheaded by The Data Lab – Scotland’s innovation centre for data and artificial intelligence (AI), the Oil & Gas Innovation Centre (OGIC) and the software company HyperDAP Ltd.
The venture, triggered in response to the data issues industries are facing, launched in 2019 and was conducted at the University of Aberdeen which supported the software research and development process for over a year.
According to the team of experts, industries are struggling to handle, process and interpret increasing quantities of data and, as a result, many data-intensive industrial sectors are at risk of failing to extract valuable information from their production and other experimentally gathered datasets.
Data has become one of the most important assets a business can own, but without proper examination and interpretation ‘it has very little value.’
The IDQI theory has since been tested, verified and validated using a very large oil and gas industrial production and monitoring dataset. It boasts three major benefits.
Firstly, it enables the use of numerical data from past research, development and production which is not currently usable because its poor quality would lead to unreliable or ‘meaningless’ results.
Secondly, it achieves massive time and cost savings by reusing existing datasets, as opposed to acquiring new datasets once again.
Finally, it allows the extraction of further knowledge and information previously masked by the high level of noise and uncertainty that is characteristic of low-quality data.
Gillian Docherty, chief executive of The Data Lab, said: “This wonderful project is a great example of how data, without processing and interpretation, can go to waste. This doesn’t just apply to the oil and gas sector and I would encourage industry leaders from any sector to get in touch with The Data Lab to discover how data can enhance your research and development work.
“Congratulations to everyone at HyperDap and the University of Aberdeen for their tenacity in seeing this project come to fruition. I look forward to seeing how IDQI is developed further.”
Case study: ‘The IDQI solution’
HyperDap, an Aberdeen-based software company, invested in IDQI technology to address the issue of numerical data quality, both in the oil and gas industry, and in many other industrial sectors.
The HyperDAP team is a combination of experienced and young professionals sharing ‘substantial expertise’ in AI and full stack development – ‘ideally equipped’ to take the project and its underpinning software technology to the next level.
The innovation and uniqueness of the IDQI is threefold:
- It will measure numerical quality, proposing strategies to improve it if this falls below a given user-defined, context-dependent level.
- It will evaluate the effect of data quality improvement techniques, proposing alternative ones if dataset quality after improvement still falls short of its target.
- It will learn about user decisions in relation to proposed data quality improvement strategies, using this knowledge to guide and improve subsequent strategies.
These innovations will give IDQI algorithm users the possibility of optimising the quality of existing production datasets in context and of assessing the impact of different computational workflows using their results, enabling a ‘much wider’ reuse of data than currently is the case.
IDQI uses software technologies based on AI in the management of massive datasets to overcome the limitations of current systems. An example of this is the lack of automated data quality evaluation and improvement functionalities and the automated suggestion of optimal workflows based on actual data quality.
The platform will address data quality and its improvement during analysis, focusing on the intelligent specification of different workflows to get meaningful results. It will automatically learn user decisions on machine-proposed workflows, using this knowledge to guide the optimal specification of subsequent workflows.
This innovation uses advanced AI techniques to perform the following:
- Improve production data quality and data analysis.
- Give users the possibility of running a number of different workflows in real time.
- Exploit machine suggestions to identify optimal solutions.
It will allow users to directly specify, program, and run their own workflows without depending on third parties and to exploit machine suggestions at their best. Current systems do not currently include automated mechanisms to evaluate and improve data quality, to recommend alternative workflows depending on evaluation results, or to learn user decisions and use them in the subsequent machine-generated proposal of further optimal actions in this context.
There is indirect positive environmental impact in the oil and gas sector that has been used as a testbed. The technology could help various leading-edge applications such as those analysing oil well integrity and performance to improve safety and decrease the likelihood of malfunctions.
Operators will also be able to identify anomalies that generate environmental pollution from the subsurface into sea water.
Many sectors currently produce large numerical datasets daily, ranging from gigabytes to terabytes. The oil and gas sector is just one such producer.
And according to the collaborators, the IDQI project has the potential to ‘revolutionise data interpretation across all sectors, bringing greater insight than ever before.’
The concept was subject to an independent peer review as part of the Oil and Gas Innovation Centre approval process, where industry experts are asked to comment on the innovativeness and applicability of the project.
One reviewer wrote: “Hydrocarbon accounting is an area of business in which small changes in effectiveness of data handling and systems quality can have a disproportionately large impact on financial gain / risk reduction / accounting robustness.
“As such, it is an ideal area for investment and the application of new technology in pursuit of those business gains. This area of business is also characterised by a large dominance of bespoke systems of very different vintages and technologies, and lack of competition in the service sector providing the niche but essential specialist services in this field.
“Thus, bringing in new ideas, new technology and new players in the service sector will be of benefit. Bringing in the latest computer science technology from academia has the potential to disrupt this market which is likely to be in need of some innovation. Good luck with the project. I look forward to hearing of its successes.”
Why innovation and marketing are the perfect partners to make changes that matter￼
With the rapid evolution of traditional marketing and the appearance of digital marketing, technology and innovation has become part of any marketer’s life without the need of working for a…
Transitioning to a four-day week – CEO’s vow to strike a healthier balance in the workplace
I came to Scotland nearly 20 years ago from Ireland, with no contacts but a lot of determination. While Ireland will always be my home, Scotland has given me amazing…
Women Lead: The female-led company championing intuitive working
Over the last two years, the pandemic forced a shift to more remote and flexible working practices. Whilst we might be seeing a “return to normal”, some companies are choosing…
Women Lead: My passion for young people to consider a career in digital
Twenty years ago, I stumbled across my career in digital marketing almost by accident. It was during my honours degree in marketing at Glasgow Caledonian University. I was on work…
Women Lead: Inclusive Silicon Valley cohort gives hope to entrepreneurs from diverse backgrounds
Things are happening on the Scottish tech scene. Big and small initiatives are creating a fantastic ripple effect on the sector, bottom up and top down, thanks to the recommendations…
Women Lead: The story of an entrepreneurial scientist
I first arrived in Scotland over 20 years ago. I had £75 in my wallet and a scholarship offer to do a PhD at the University of Edinburgh. Sometimes I…
Please mind the gap… or healthcare may fall
Imagine sharing a lengthy train journey with others. From beginning to end, imagine how often you might hear ‘mind the gap’ messages about embarking and disembarking safely. Picture how navigating…
Women Lead: My journey from Dragons’ Den to Silicon Valley
Following her appearance on Dragons’ Den, Sheila Hogan, serial entrepreneur, founder and chief executive of digital legacy vault, Biscuit Tin, shares her experience of her time in the Den and…