Tag Archives: prediction

New statistical approach aims to predict when low-probability, high-impact events happen

A team of researchers from the U.S. and Hong Kong is working to develop new methods of statistical analysis that may let us predict the risks of very rare but dramatic events such as pandemics, earthquakes, or meteorite strikes happening in the future.

Image via Pixabay.

Our lives over these last two years have been profoundly marked by the pandemic — and although researchers warned us about the risk of a pandemic, society was very much surprised. But what if we could statistically predict the risk of such an event happening in advance?

An international team of researchers is working towards that exact goal by developing a whole new way to perform statistical analyses. Typically, events of such rarity are very hard to study through the prism of statistical methods, as they simply happen too rarely to yield reliable conclusions.

The method is in its early stages and, as such, hasn’t proven itself. But the team is confident that their work can help policymakers better prepare for world-spanning, dramatic events in the future.

Black swans

“Though they are by definition rare, such events do occur and they matter; we hope this is a useful set of tools to understand and calculate these risks better,” said mathematical biologist Joel Cohen, a professor at the Rockefeller University and at the Earth Institute of Columbia University, and a co-author of the study describing the findings.

The team hopes that their work will give statisticians an effective tool with which to analyze sets of data when it contains very sparse points of data, as is the case for very dramatic (positive or negative) events. This, they argue, would give government officials and other decision-makers a way to make informed decisions when planning for such events in the future.

Statistics by now is a tried and true field of mathematics. It’s one of our best tools when trying to make sense of the world around us and, generally, serves us well. However, the quality of the conclusions statistics can draw from a dataset relies directly on how rich those datasets are, and the quality of the information they contain. As such, statistics has a very hard time dealing with events that are exceedingly rare.

That hasn’t stopped statisticians from trying to apply their methods to rare-but-extreme events, however, over the last century or so. It’s still a relatively new field of research in the grand scheme of things, so we’re still learning what works here and what doesn’t. Where a worker would need to use the appropriate tool for the job at hand, statisticians need to apply the right calculation method on their dataset; which method they employ has a direct impact on which conclusions they draw, and how reliably these reflect reality.

Two important parameters when processing a dataset are the average value and the variance. You’re already familiar with what an average value is. The variance, however, shows how far apart the values that make up that average are. For example, both 0 and 100, as well as 49 and 51, average out to 50; the first set, however, has a much larger variance than the latter.

Black swan theory describes events that come as a surprise but have a major effect and then are inappropriately rationalized after the fact with the benefit of hindsight. The new research doesn’t only focus on black swans, but on all unlikely events that would have a major impact.

For typical sets, the average value and the variance can both be defined by finite numbers. In the case of the events that made the object of this study, however, the sheer rarity with which they take place can push these numbers towards ridiculous values bordering on infinity. World wars, for example, have been extremely rare events in human history, but each one has also had an incredibly large effect, shaping the world into what it is today.

“There’s a category where large events happen very rarely, but often enough to drive the average and/or the variance towards infinity,” said Cohen.

Such datasets require new tools to be properly handled, the team argues. If we can make heads and tails of it, however, we could be much better prepared for them, and see a greater return on investments into preparedness. Governments and other ruling bodies would obviously stand to benefit from having such information on hand.

Being able to accurately predict the risk of dramatic events would also benefit us as individuals, and provide important tangible benefits in society. From allowing us better plan out our lives (who here wouldn’t have liked to know that the pandemic was going to happen in advance?), to better preparing for threatening events, to giving us arguments for lower insurance premiums, such information would definitely be useful to have. If nothing bad is likely to happen during our lifetimes, you could argue, wouldn’t it make sense for my life insurance policy premiums to be lower? The insurance industry in the US alone is worth over $1 trillion and making the system more efficient could amount to major savings.

But does it work?

The authors started from mathematical models used to calculate risk and examined whether they can be adapted to analyze low-probability, very high-impact events with infinite mean and variance. The standard approach these methods use involves semi-variances: the practice of separating the dataset in ‘below-average’ and ‘above-average’ halves, then examining the risk in each. Still, this didn’t provide reliable data.

What does work, the authors explain, is to examine the log (logarithmic function) of the average to the log of the semi-variance in each half of the dataset. Logarithmic functions are the reverse of exponentials, just like division is the reverse of multiplication. They’re a very powerful tool when you’re dealing with massive, long numbers, as they simplify the picture without cutting out any meaningful data — ideal for studying the kind of numbers produced by rare events.

“Without the logs, you get less useful information,” Cohen said. “But with the logs, the limiting behavior for large samples of data gives you information about the shape of the underlying distribution, which is very useful.”

While this study isn’t the end-all-be-all of the topic, it does provide a strong foundation for other researchers to build upon. For now, although new and in their infancy, the findings do hold promise. Right now, they’re the closest we’ve gotten to a formula that can predict when something big is going to happen.

“We think there are practical applications for financial mathematics, for agricultural economics, and potentially even epidemics, but since it’s so new, we’re not even sure what the most useful areas might be,” Cohen said. “We just opened up this world. It’s just at the beginning.”

The paper Taylor’s law of fluctuation scaling for semivariances and higher moments of heavy-tailed data” has been published in the journal Proceedings of the National Academy of Sciences.

Atmospheric CO2 levels this year could reach 150% of those before the Industrial Revolution

Our climate is changing, and the cause is our own emissions. To put those into perspective, new research estimates that atmospheric CO2 levels in 2021 will be 50% higher than the average value in the 18th century (the onset of the Industrial Revolution).

Image credits Chris LeBoutillier.

The Met Office, Britain’s national weather service, estimates in a new report that average annual CO2 levels this year (as measured at the Mauna Loa Observatory in Hawaii), will rise by roughly 2.29 ppm (parts per million) compared to 2020. That is around 150% of the concentration this gas registered in the 18th century, before industrial emissions started to output in significant quantities.

Bad air

“Since CO2 stays in the atmosphere for a very long time, each year’s emissions add to those from previous years and cause the amount of CO2 in the atmosphere to keep increasing,” said Richard Betts, lead producer of the Met Office’s annual CO2 forecast.

The most worrying observation is that CO2 levels are still expected to rise in 2021 despite a significant drop in total emission levels due to the pandemic.

Mauna Loa is used as a gold-standard for the measurement of CO2 levels in the atmosphere. The site has been in operation monitoring this gas since 1958. These show seasonal variation , but they’re also influenced by local factors and geography, so having a single monitoring point in operation for so long makes the readings more reliable, as they can be easily compared to past readings.

Still, the news leaves us in an unenviable spot. According to the United Nations, emissions from energy, food production, transport and industry must drop by 7% per year every year throughout the next decade if we’re to meet the target of the Paris climate deal. This international deal aims to keep global warming “well below” 2 degrees Celsius (3.6 degrees Fahrenheit) above pre-industrial levels — ideally below 1.5 degrees Celsius.

We’ve only seen 1 degree Celsius of warming (compared to pre-industrial levels) so far, and yet, we have seen extreme weather events such as floods, droughts, and tropical storms pick up around the globe. The seas are also rising to meet us.

According to the Met Office, it took 200 years for atmospheric CO2 concentrations to rise 25% above pre-industrial levels; it only took an extra 30 to get to double that. It might take even less to double that figure yet again unless we take serious action — and do so quickly.

“Reversing this trend and slowing the atmospheric CO2 rise will need global emissions to reduce, and bringing them to a halt will need global emissions to be brought down to net zero.”

Google AI can now look at your retina and predict the risk of heart disease

Google researchers are extremely intuitive: just by looking into people’s eyes they can see their problems — cardiovascular problems, to be precise. The scientists trained artificial intelligence (AI) to predict cardiovascular hazards, such as strokes, based on the analysis of retina shots.

The way the human eye sees the retina vs the way the AI sees it. The green traces are the pixels used to predict the risk factors. Photo Credit: UK Biobank/Google

After analyzing data from over a quarter million patients, the neural network can predict the patient’s age (within a 4-year range), gender, smoking status, blood pressure, body mass index, and risk of cardiovascular disease.

“Cardiovascular disease is the leading cause of death globally. There’s a strong body of research that helps us understand what puts people at risk: Daily behaviors including exercise and diet in combination with genetic factors, age, ethnicity, and biological sex all contribute. However, we don’t precisely know in a particular individual how these factors add up, so in some patients, we may perform sophisticated tests … to help better stratify an individual’s risk for having a cardiovascular event such as a heart attack or stroke”, declared study co-author Dr. Michael McConnell, a medical researcher at Verily.

Even though you might think that the number of patients the AI was trained on is large, AI networks typically work with much larger sample sizes. In order for neural networks to be more accurate in their predictions, they must analyze as much data as possible. The results of this study show that, until now, the predictions made by AI cannot outperform specialized medical diagnostic methods, such as blood tests.

“The caveat to this is that it’s early, (and) we trained this on a small data set,” says Google’s Lily Peng, a doctor and lead researcher on the project. “We think that the accuracy of this prediction will go up a little bit more as we kind of get more comprehensive data. Discovering that we could do this is a good first step. But we need to validate.”

The deep learning applied to photos of the retina and medical data works like this: the network is presented with the patient’s retinal shot, and then with some medical data, such as age, and blood pressure. After seeing hundreds of thousands of these kinds of images, the machine will start to see patterns correlated with the medical data inserted. So, for example, if most patients that have high blood pressure have more enlarged retinal vessels, the pattern will be learned and then applied when presented just the retinal shot of a prospective patient. The algorithms correctly discovered patients who had great cardiovascular risks within a 5-year window 70 percent of the time.

“In summary, we have provided evidence that deep learning may uncover additional signals in retinal images that will allow for better cardiovascular risk stratification. In particular, they could enable cardiovascular assessment at the population level by leveraging the existing infrastructure used to screen for diabetic eye disease. Our work also suggests avenues of future research into the source of these associations, and whether they can be used to better understand and prevent cardiovascular disease,” conclude the authors of the study.

The paper, published in the journal Nature Biomedical Engineering, is truly remarkable. In the future, doctors will be able to screen for the number one killer worldwide much more easily, and they will be doing it without causing us any physical discomfort. Imagine that!

Scientists claim they have identified a 'crystal ball' mathematical equation which can be used to predict if a system is about to move over to a disorderly state. In theory, it could be used to predict complex real life systems like financial stock market crashes.

Mathematical equation helps predict calamities, financial crashes or epilepsy seizures

Scientists claim they have identified a 'crystal ball' mathematical equation which can be used to predict if a system is about to move over to a disorderly state. In theory, it could be used to predict complex real life systems like financial stock market crashes.

Scientists claim they have identified a ‘crystal ball’ mathematical equation which can be used to predict if a system is about to move over to a disorderly state. In theory, it could be used to predict complex real life systems like financial stock market crashes.

In science we have what are called “laws”, be them Newton’s Laws of Motion or Archimedes’ Principle, because these mathematical expressions describe systems in a rigid set of boundaries, essentially helping predict how these systems will behave in the future. What about overly complex, highly dynamic systems; could we use a single mathematical equation to predict outcomes for such systems? An  University of Sussex-led study found a mathematical equation that may help predict calamities such as financial crashes in economic systems and epileptic seizures in the brain.

The team of neuroscientists led by Dr Lionel Barnett sought to mathematically describe how various parts of a systems simultaneously behave differently, while still being integrated (the parts depend on each other). Collaborating with scientists at the University’s Sackler Centre for Consciousness Science and the Centre for Research in Complex Systems at Charles Sturt University in Australia, the team used mathematics and detailed computer simulations to show that a measure of ‘information flow’ reaches a peak just before a system moves from a healthy state to an unhealthy state.

This is known as a ‘phase transition’ and in real world systems these can have huge implications, like epileptic seizures or financial market crashes. Predicting such events in the past had been extremely difficult to undertake. Barnett and colleagues, however, showed for the first time that their method can reliably predict phase transitions in a physics standard system – so-called Ising models.

” This conjecture is verified for a ferromagnetic 2D lattice Ising model with Glauber dynamics and a transfer entropy-based measure of systemwide information flow. Implications of the conjecture are considered, in particular, that for a complex dynamical system in the process of transitioning from disordered to ordered dynamics (a mechanism implicated, for example, in financial market crashes and the onset of some types of epileptic seizures); information dynamics may be able to predict an imminent transition,” reads the paper’s abstract.

“The key insight in the paper is that the dynamics of complex systems – like the brain and the economy – depend on how their elements causally influence each other; in other words, how information flows between them. And that this information flow needs to be measured for the system as a whole, and not just locally between its various parts,” Dr. Barnett said.

It will be interesting to see how University of Susses researchers’ method fairs with complex real world system, and to which degree their equation can reliably predict when a phase transition will occur.

Professor Anil Seth, Co-Director of the Sackler Centre, says: “The implications of the work are far-reaching. If the results generalise to other real-world systems, we might have ways of predicting calamitous events before they happen, which would open the possibility for intervention to prevent the transition from occurring.”
“For example, the ability to predict the imminent onset of an epileptic seizure could allow a rapid medical intervention (perhaps via brain stimulation) which would change the course of the dynamics and prevent the seizure. And if similar principles apply to financial markets, climate systems, and even immune systems, similar interventions might be possible. Further research is needed to explore these exciting possibilities.”

The findings were published in the journal Physical Review Letters.