When we launched CureTogether in 2008, we believed that if we could simply ask thousands of people about the details of their experience with different health conditions; gather their responses in a structured, quantitative way; and apply the right algorithms to that data; important discoveries would emerge that could reduce needless suffering.
Considering how valuable good health is to all of us, and how much time and money is spent each year to maintain it, it seems incredibly wasteful for millions of people to leave, locked in their heads, detailed knowledge about their experience with conditions, symptoms and treatments.
Migraine was one of the first conditions added to CureTogether, and one for which we’ve gathered the most data. One of the most popular and highest rated treatments is Imitrex (Sumatriptan). Most people who try Imitrex report an improvement in their symptoms, but a significant 11% report feeling “much worse” after taking it.
There are two good reasons why someone with Migraine would want to know how their risk of having a negative reaction compares to the average: 1) Migraines are hard enough to endure on their own – you don’t want a negative reaction to a treatment to make it even harder and 2) Imitrex is expensive – costing as much as $30 per tablet!
We were interested in whether differences in symptoms might account for some people reacting negatively to a treatment that otherwise helped so many people feel better. So, starting with a subset of 38 Migraine symptoms that at least 100 members had reported having, and at least 100 members had reported not having, and for which all members had also provided effectiveness ratings for Imitrex, we segmented members into two groups:
Group A = those who reported having the symptom
Group B = those who reported not having the symptom
We then further segmented these members according to how they rated Imitrex:
- major improvement
- moderate improvement
- no effect or uncertain
- made it slightly worse
- made it much worse
To determine which symptoms had the greatest influence on negative ratings of Imitrex, we calculated, for each row, the negative response ratio (NRR) as follows:
NRR = p(MW|Y) / p(MW|N)
- p(MW|Y) = the fraction of members having the symptom who rated Imitrex “much worse”
- p(MW|N) = the fraction of members NOT having the symptom who rated Imitrex “much worse”
After ranking the symptoms from highest to lowest NRR, ”Vertigo / Dizziness” came out on top, with an NRR of 4.2x.
Among CureTogether members with Migraine, the overall probability of reporting a “much worse” response to Imitrex is 11%. Of those who reported not having “Vertigo / Dizziness”, only 4.0% reported a “much worse” response to Imitrex. Of those who reported having “Vertigo / Dizziness”, 17% reported a “much worse” response. Thus, members who reported “Vertigo / Dizziness” were 4.2x more likely to report a “much worse” response.
To test for significance, we needed to determine the probability that a 4.2x (or more extreme) NRR could be generated by chance, rather than as a result of a genuine influence. Our null hypothesis was that whether someone had, or did not have, Vertigo / Dizziness had no effect on Imitrex ratings. We generated 1 million fake datasets by randomly permuting the User ID’s for Imitrex ratings, pairing members of Groups A and B with real, but random Imitrex ratings, rather than their own, and measuring the NRR each time.
The frequency of a 4.2x NRR within the fake datasets would indicate the probability that the such a value could be generated by chance, assuming the null hypothesis were true. We ran the analysis and plotted the sampling distribution of 1 million NRRs as a histogram:
An NRR of 4.2x would be generated with a probability of 2.44e-4 – i.e. only one in ~4,000 times would we see such a result by chance if the null hypothesis were true. Usually, scientists consider a p-value of < 0.05 to be significant, so this is a very significant result, even if you factor in the fact that we only discovered the 4.2x result after sorting 38 symptoms.
But dizziness is a known side-effect of Imitrex. How do we know that those CureTogether members who tried Imitrex were not just more likely to report dizziness as a side-effect, rather than as a symptom prior to trying Imitrex? How do we know that Vertigo / Dizziness actually predicts response to Imitrex? Well, if it were a side-effect, then we would expect members who tried Imitrex to be more likely to report significantly more Vertigo/Dizziness. Here’s what the data says:
While 76% of those who tried Imitrex (224/293) reported Vertigo / Dizziness, 72% of those who did NOT try Imitrex (246/340) also reported it. There is only a 5% difference between these two groups, in contrast to the 4.2x difference above. While this may seem counterintuitive, keep in mind that the probability of reporting a symptom, given that you have tried a treatment is not the same thing as the probability of giving a treatment a negative rating, given that you’ve had the symptom. p(s|t) != p(mw|s).
Now, there is one caveat – we cannot assume that these results apply to all Migraine sufferers. Although we see promising early indications that CureTogether members are representative of the general population, we are not ready to make that claim conclusively.
Thus, we have demonstrated that, among CureTogether members with Migraine at least, having Vertigo / Dizziness as a symptom increases the chance of reporting a “much worse” response to Imitrex by 4.2x, from 4.0% to 17%.
We are enthusiastic to have this result confirm our initial belief in the power of CureTogether’s approach to health research. We also believe this is the first of many predictive discoveries across the 500+ conditions for which we continue to gather data.
Thank you to our members who took the time to participate in our surveys. And special thanks to Dr. Will Dampier, Research Assistant Professor at Drexel University School of Biomedical Engineering and Health Systems, for his help with the statistical analysis.