Re-Analysis of Randomized Controlled Trials
Re-Analysis of Randomized Controlled Trials
l
When testing the efficacy of a treatment or intervention, randomized controlled trials (RCT) are considered the gold standard in study design. Despite randomization’s ability to decrease bias, treatment contamination—an issue that arises when both groups in a trial are exposed to the intervention or active treatment— is difficult to account for, especially when the intervention is behavioral in nature.
Despite several randomized controlled trials (RCT) showing an association between AA involvement and better alcohol-related outcomes, this self selection bias makes it difficult to determine if benefits are inherent to AA’s effectiveness or rather an artifact of the bias. AA meeting attendance of people randomized to the AAFI condition is influenced by both randomization (i.e., exposure to an intervention encouraging meeting attendance) and self-selection factors (e.g., motivation to change) while people randomized to the non-Alcoholics Anonymous Facilitation Interventions (AAFI) condition are solely influenced by these self-selection factors.
Humphreys and colleagues used instrumental variables to correct for self selection bias through reanalysis of six randomized controlled trials (RCT) datasets to determine if AA attendance (due to treatment assignment and free from selection-bias) improved percent days abstinent (PDA) at 3 and 15 months in people with alcohol use disorder. This method uses an external variable (i.e., randomized treatment assignment) that makes the factor under study (AA meeting attendance) more or less likely to occur but is not directly associated with the outcome (PDA).
The effect of treatment assignment on percent days abstinent (PDA) determined through traditional intention-to-treat analysis is adjusted by treatment non-adherence. In other words, the treatment effects are adjusted for the percentage of participants assigned to Alcoholics Anonymous Facilitation Interventions (AAFI) who do not attend 12-step meetings and the percentage of participants assigned to non-AAFI who do attend meetings. The authors found that five of the six datasets were statistically similar, allowing for a pooled analysis; the other dataset (from the aftercare arm of Project MATCH which was the only inpatient sample) was analyzed separately. In the pooled sample, 79% of participants were white, 72% were male, and 51% were ages 35-54. By comparison, the MATCH aftercare participants were older (72% over age 35 vs. 62% in pooled sample) and more likely to be male (80%). At baseline, the MATCH aftercare sample had lower average PDA than the pooled sample (27 vs. 38, respectively) and higher average percent days attending a meeting (PDMT) (4 vs. 3, respectively).
Using the instrumental variable models to determine the impact of additional AA meeting attendance due to random assignment (i.e., meetings attended beyond what the individual would have selected to attend in the absence of the intervention), the authors found that in the pooled sample, an increase in PDMT was significantly associated with increased percent days abstinent (PDA) at 3 and 15 months. At 3 months, an increase in attendance of 1% predicated an increase in PDA of one third of a percentage point. In simpler terms, going to an additional 2 AA meetings per week (defined as an increase in PDMT of 28.6 absolute percentage points) results in a gain of 3.3 days of abstinence per month (increase in PDA of 10.9 percentage points).
At 15 months, the association between percent days attending a meeting (PDMT) and percent days abstinent (PDA) was even stronger. For the MATCH aftercare sample, no association between percent days attending a meeting (PDMT) and percent days abstinent (PDA) was found suggesting there is no benefit in attending additional AA meetings beyond what participants already attended without the intervention.
These results suggest that for people seeking treatment, increasing AA meeting attendance can have a beneficial impact on abstinence, a result that is directly attributable to AA attendance itself.
The lack of association from the Project MATCH sample may be explained by the fact that this population had higher levels of pre-existing AA involvement. Thus, those achieving abstinence due to AA will not become anymore abstinent if they increase their meeting attendance and those with high attendance who have not benefitted are unlikely to benefit from adding even more attendance.
Statistical methods provide a powerful tool for obtaining the most bias-free estimates when study design alone is unable to prevent all threats to validity. For instrumental variable modeling, it is crucial that the instrumental variable used is a strong predictor of the outcome. In this study, random treatment assignment was a strong predictor of percent days attending a meeting (PDMT), making this method possible. Researchers experiencing similar issues due to treatment contamination and self-selection bias should consider using these methods.
Since this article focused on patients with alcohol use disorders and AA attendance, more work is needed on patients with drug use disorders and on other 12-step mutual-help organizations like Narcotics Anonymous.
Humphreys, K., Blodgett, J. C., & Wagner, T. H. (2014). Estimating the efficacy of Alcoholics Anonymous without self-selection bias: an instrumental variables re-analysis of randomized clinical trials. Alcohol Clin Exp Res, 38(11), 2688-2694. doi: 10.1111/acer.12557
l
When testing the efficacy of a treatment or intervention, randomized controlled trials (RCT) are considered the gold standard in study design. Despite randomization’s ability to decrease bias, treatment contamination—an issue that arises when both groups in a trial are exposed to the intervention or active treatment— is difficult to account for, especially when the intervention is behavioral in nature.
Despite several randomized controlled trials (RCT) showing an association between AA involvement and better alcohol-related outcomes, this self selection bias makes it difficult to determine if benefits are inherent to AA’s effectiveness or rather an artifact of the bias. AA meeting attendance of people randomized to the AAFI condition is influenced by both randomization (i.e., exposure to an intervention encouraging meeting attendance) and self-selection factors (e.g., motivation to change) while people randomized to the non-Alcoholics Anonymous Facilitation Interventions (AAFI) condition are solely influenced by these self-selection factors.
Humphreys and colleagues used instrumental variables to correct for self selection bias through reanalysis of six randomized controlled trials (RCT) datasets to determine if AA attendance (due to treatment assignment and free from selection-bias) improved percent days abstinent (PDA) at 3 and 15 months in people with alcohol use disorder. This method uses an external variable (i.e., randomized treatment assignment) that makes the factor under study (AA meeting attendance) more or less likely to occur but is not directly associated with the outcome (PDA).
The effect of treatment assignment on percent days abstinent (PDA) determined through traditional intention-to-treat analysis is adjusted by treatment non-adherence. In other words, the treatment effects are adjusted for the percentage of participants assigned to Alcoholics Anonymous Facilitation Interventions (AAFI) who do not attend 12-step meetings and the percentage of participants assigned to non-AAFI who do attend meetings. The authors found that five of the six datasets were statistically similar, allowing for a pooled analysis; the other dataset (from the aftercare arm of Project MATCH which was the only inpatient sample) was analyzed separately. In the pooled sample, 79% of participants were white, 72% were male, and 51% were ages 35-54. By comparison, the MATCH aftercare participants were older (72% over age 35 vs. 62% in pooled sample) and more likely to be male (80%). At baseline, the MATCH aftercare sample had lower average PDA than the pooled sample (27 vs. 38, respectively) and higher average percent days attending a meeting (PDMT) (4 vs. 3, respectively).
Using the instrumental variable models to determine the impact of additional AA meeting attendance due to random assignment (i.e., meetings attended beyond what the individual would have selected to attend in the absence of the intervention), the authors found that in the pooled sample, an increase in PDMT was significantly associated with increased percent days abstinent (PDA) at 3 and 15 months. At 3 months, an increase in attendance of 1% predicated an increase in PDA of one third of a percentage point. In simpler terms, going to an additional 2 AA meetings per week (defined as an increase in PDMT of 28.6 absolute percentage points) results in a gain of 3.3 days of abstinence per month (increase in PDA of 10.9 percentage points).
At 15 months, the association between percent days attending a meeting (PDMT) and percent days abstinent (PDA) was even stronger. For the MATCH aftercare sample, no association between percent days attending a meeting (PDMT) and percent days abstinent (PDA) was found suggesting there is no benefit in attending additional AA meetings beyond what participants already attended without the intervention.
These results suggest that for people seeking treatment, increasing AA meeting attendance can have a beneficial impact on abstinence, a result that is directly attributable to AA attendance itself.
The lack of association from the Project MATCH sample may be explained by the fact that this population had higher levels of pre-existing AA involvement. Thus, those achieving abstinence due to AA will not become anymore abstinent if they increase their meeting attendance and those with high attendance who have not benefitted are unlikely to benefit from adding even more attendance.
Statistical methods provide a powerful tool for obtaining the most bias-free estimates when study design alone is unable to prevent all threats to validity. For instrumental variable modeling, it is crucial that the instrumental variable used is a strong predictor of the outcome. In this study, random treatment assignment was a strong predictor of percent days attending a meeting (PDMT), making this method possible. Researchers experiencing similar issues due to treatment contamination and self-selection bias should consider using these methods.
Since this article focused on patients with alcohol use disorders and AA attendance, more work is needed on patients with drug use disorders and on other 12-step mutual-help organizations like Narcotics Anonymous.
Humphreys, K., Blodgett, J. C., & Wagner, T. H. (2014). Estimating the efficacy of Alcoholics Anonymous without self-selection bias: an instrumental variables re-analysis of randomized clinical trials. Alcohol Clin Exp Res, 38(11), 2688-2694. doi: 10.1111/acer.12557
l
When testing the efficacy of a treatment or intervention, randomized controlled trials (RCT) are considered the gold standard in study design. Despite randomization’s ability to decrease bias, treatment contamination—an issue that arises when both groups in a trial are exposed to the intervention or active treatment— is difficult to account for, especially when the intervention is behavioral in nature.
Despite several randomized controlled trials (RCT) showing an association between AA involvement and better alcohol-related outcomes, this self selection bias makes it difficult to determine if benefits are inherent to AA’s effectiveness or rather an artifact of the bias. AA meeting attendance of people randomized to the AAFI condition is influenced by both randomization (i.e., exposure to an intervention encouraging meeting attendance) and self-selection factors (e.g., motivation to change) while people randomized to the non-Alcoholics Anonymous Facilitation Interventions (AAFI) condition are solely influenced by these self-selection factors.
Humphreys and colleagues used instrumental variables to correct for self selection bias through reanalysis of six randomized controlled trials (RCT) datasets to determine if AA attendance (due to treatment assignment and free from selection-bias) improved percent days abstinent (PDA) at 3 and 15 months in people with alcohol use disorder. This method uses an external variable (i.e., randomized treatment assignment) that makes the factor under study (AA meeting attendance) more or less likely to occur but is not directly associated with the outcome (PDA).
The effect of treatment assignment on percent days abstinent (PDA) determined through traditional intention-to-treat analysis is adjusted by treatment non-adherence. In other words, the treatment effects are adjusted for the percentage of participants assigned to Alcoholics Anonymous Facilitation Interventions (AAFI) who do not attend 12-step meetings and the percentage of participants assigned to non-AAFI who do attend meetings. The authors found that five of the six datasets were statistically similar, allowing for a pooled analysis; the other dataset (from the aftercare arm of Project MATCH which was the only inpatient sample) was analyzed separately. In the pooled sample, 79% of participants were white, 72% were male, and 51% were ages 35-54. By comparison, the MATCH aftercare participants were older (72% over age 35 vs. 62% in pooled sample) and more likely to be male (80%). At baseline, the MATCH aftercare sample had lower average PDA than the pooled sample (27 vs. 38, respectively) and higher average percent days attending a meeting (PDMT) (4 vs. 3, respectively).
Using the instrumental variable models to determine the impact of additional AA meeting attendance due to random assignment (i.e., meetings attended beyond what the individual would have selected to attend in the absence of the intervention), the authors found that in the pooled sample, an increase in PDMT was significantly associated with increased percent days abstinent (PDA) at 3 and 15 months. At 3 months, an increase in attendance of 1% predicated an increase in PDA of one third of a percentage point. In simpler terms, going to an additional 2 AA meetings per week (defined as an increase in PDMT of 28.6 absolute percentage points) results in a gain of 3.3 days of abstinence per month (increase in PDA of 10.9 percentage points).
At 15 months, the association between percent days attending a meeting (PDMT) and percent days abstinent (PDA) was even stronger. For the MATCH aftercare sample, no association between percent days attending a meeting (PDMT) and percent days abstinent (PDA) was found suggesting there is no benefit in attending additional AA meetings beyond what participants already attended without the intervention.
These results suggest that for people seeking treatment, increasing AA meeting attendance can have a beneficial impact on abstinence, a result that is directly attributable to AA attendance itself.
The lack of association from the Project MATCH sample may be explained by the fact that this population had higher levels of pre-existing AA involvement. Thus, those achieving abstinence due to AA will not become anymore abstinent if they increase their meeting attendance and those with high attendance who have not benefitted are unlikely to benefit from adding even more attendance.
Statistical methods provide a powerful tool for obtaining the most bias-free estimates when study design alone is unable to prevent all threats to validity. For instrumental variable modeling, it is crucial that the instrumental variable used is a strong predictor of the outcome. In this study, random treatment assignment was a strong predictor of percent days attending a meeting (PDMT), making this method possible. Researchers experiencing similar issues due to treatment contamination and self-selection bias should consider using these methods.
Since this article focused on patients with alcohol use disorders and AA attendance, more work is needed on patients with drug use disorders and on other 12-step mutual-help organizations like Narcotics Anonymous.
Humphreys, K., Blodgett, J. C., & Wagner, T. H. (2014). Estimating the efficacy of Alcoholics Anonymous without self-selection bias: an instrumental variables re-analysis of randomized clinical trials. Alcohol Clin Exp Res, 38(11), 2688-2694. doi: 10.1111/acer.12557
SIGN UP FOR THE BULLETIN
151 Merrimac St., 4th Floor. Boston, MA 02114
