Big health data should be on sale in supermarkets. It offers great promise to inform our future health. To be able to assess the benefits of different treatments for each of us as individuals – both in the short and long term would be good shopping – if only we could access and read the labels.

Data mining has to go public.

At the moment, big datasets are only available to warehousers. In the US, they are little more than financial instruments used to assess prescription drug use, hospital expenditure and predict generalized treatment pathways. In the UK, the major data set, Clinical Practice Research Datalink can link the hospital, cancer and social deprivation datasets to primary care consultations – for health researchers only.

Even these enormous datasets have limitations for those who can access them.

Large data sets are enormous and unruly beasts that need a lot of grooming to make them fit for any purpose – let alone access by individuals. Accessing them is still time consuming and often thankless work for those with aggregation of data in mind.  Even more frustrating for us would be to find out that the questions we want answered can’t be answered, usually because the data is collected for fiscal reporting not treatment and often out of date. Most datasets are retrospective and usually somewhere between 6 months and several years out of date. They are static and not made to be accessible in real time nor interactive. This makes planning difficult, especially in the fields of medicine which are rapidly changing such as oncology.

Newer techniques of data mining hold vast potential. Mining has the capacity to go beyond asking questions. It can make links in the data that are not obvious to researchers and, from these connections, treatment pathways can be developed.

This is the promise. However, there are still major hurdles to be overcome. Individuals can get lost in aggregation. For example, there are be over 3 million genetic variations between two individuals and any of these can change our response to treatment. And that’s not just for the new precision drugs. Even established drugs, eg antihypertensives, aspirin, acetaminophen, paracetamol and antibiotics, work differently in each of us.

Data mining has to go public. It has the potential to enhance individual care if we are able to access the data and manipulate it ourselves. If this doesn’t happen it will take health care further away from the individual to where no amount of communication skills can bridge the gap.