Software and package applicating for network meta-analysis: A usage-based comparative study
Abstract
Objective
To compare and analyze the characteristics and functions of software applications for network meta-analysis (NMA).
Methods
PubMed, EMbase, The Cochrane Library, the official websites of Bayesian inference Using Gibbs Sampling (BUGS), Stata and R, and Google were searched to collect the software and packages for performing NMA; software and packages published up to March 2016 were included. After collecting the software, packages, and their user guides, we used the software and packages to calculate a typical example. All characteristics, functions, and computed results were compared and analyzed.
Results
Ten types of software were included, including programming and non-programming software. They were developed mainly based on Bayesian or frequentist theory. Most types of software have the characteristics of easy operation, easy mastery, exact calculation, or excellent graphing. However, there was no single software that performed accurate calculations with superior graphing; this could only be achieved through the combination of two or more types of software.
Conclusion
This study suggests that the user should choose the appropriate software according to personal programming basis, operational habits, and financial ability. Then, the choice of the combination of BUGS and R (or Stata) software to perform the NMA is considered.
1 INTRODUCTION
For most healthcare problems, there are a large number of competing interventions. The relative effectiveness of treatments is typically assessed in randomized control trials1, 2 (RCTs), and these RCT comparisons form a network of treatment comparisons around which information may flow as long as the network is connected. Such a body of evidence can be synthesized via network meta-analysis (NMA).3
NMA,4 also called multiple-treatments meta-analysis (MTA) or mixed-treatment comparison (MTC), is an extension of the traditional pairwise meta-analysis that enables simultaneous comparisons between multiple interventions, respecting the randomization in the evidence. NMA is now an established method in the evidence-based medicine literature and is being increasingly used to assess the comparative effectiveness of healthcare interventions.4, 5
Since NMA was formally proposed as a generalization of pairwise meta-analysis in 2002,6 its methodologies and software have been developed and applied rapidly. Based on the inherent characteristics, such as many intervention measures, large volumes of data, the complex internal structure, and so on, the dependence on NMA's software has become more prominent and imperative compared with pairwise meta-analysis. Many types of software7-9 are capable of performing calculations for NMA, but separate and comprehensive software that can perform calculations with the relevant perfect graphing is not available. Hence, multiple types of software must be combined. At present, although the latest version of the Cochrane handbook10 introduces NMA and suggests using WinBUGS software, the graph function of this software is deficient. It is challenging to choose different types of software and combine them to perform NMA efficiently. Our study compared and analyzed available software for performing NMA to provide references for users to effectively choose relevant software.
2 METHODS
2.1 Selection criteria
Software and packages have the capacity to perform data operations of NMA. They are not limited in terms of graphing ability, not limited with the programming or un-programming software, and not limited to software platform based on language.
2.2 Literature search
PubMed, EMbase, The Cochrane Library, and the official websites of BUGS (Bayesian inference Using Gibbs Sampling), Stata, R, and Google were searched and viewed. Software and packages used in the published NMAs were recorded and methodology literature introducing software and packages up to March 2016 were read. Then, these software packages and their introductions were downloaded. The MeSH and text words were retrieved: “Meta-Analyses, Network,” “Meta-Analysis, Network,” “Network Meta Analysis,” “Network Meta-Analysis,” “Network Meta-Analyses,” “Mixed Treatment Meta-Analysis,” “Meta-Analyses, Mixed Treatment,” “Meta-Analysis, Mixed Treatment,” “Mixed Treatment Meta Analysis,” “Mixed Treatment Meta-Analyses,” “Multiple Treatment Comparison Meta-Analysis,” and “Multiple Treatment Comparison Meta Analysis.”
2.3 Adopting methods
The introductions to the different software and packages are compared based on theory, model, permission settings, whether they are used for programming code input, and whether called and calling programs are involved in the software and packages. Taking the example of data related to the effectiveness of 13 antidepressants and placebo in the treatment of depression from a study,11 many types of software and packages are used to analyze and compare their calculative functions, the results, and graphing functions.
3 RESULTS
3.1 Included software
Ten types of software were included (Table 1). Free software included BUGS,12 JAGS,13 Stan,14 GeMTC,15 ADDIS,16 and R.17 Proprietary software included Microsoft Excel,18 Stata,19 and SAS.20 All software had been obtained online.
| Feature | Based on Theory | Based on Model | Computing Functions | Graphic Functions | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Software | Free | program | Bayesian | Frequency | Hierarchical model | Regression model | Multivariate analysis | Two-stage approach | Carries computing functions | Call other software for computing functions | It Can be Called? | Network plot | Forest plot | Contour-enhanced funnel plot | Contribution plot | Rank plot |
| WinBUGS | √ | √ | √ | × | √ | √ | √ | √ | √ | × | √ | × | √ | × | × | × |
| OpenBUGS | √ | √ | √ | × | √ | √ | √ | √ | √ | × | √ | × | √ | × | × | × |
| JAGS | √ | √ | √ | × | √ | √ | √ | √ | √ | × | √ | × | × | × | × | × |
| Stan | √ | √ | √ | × | √ | √ | √ | √ | √ | × | √ | × | × | × | × | × |
| Microsoft Excel | × | √ | √ | × | √ | √ | √ | √ | × | √ | √ | × | × | × | × | × |
| GeMTC | √ | × | √ | × | √ | × | × | × | √ | × | × | × | × | × | × | × |
| ADDIS | √ | × | √ | × | √ | × | × | × | √ | × | × | √ | × | × | × | × |
| R | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ |
| Stata | × | * | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ |
| SAS | × | √ | √ | √ | √ | √ | √ | √ | √ | √ | √ | × | √ | × | × | × |
- Note: √: With this feature; ×: It does not have this feature; *: With programming and non-programming functions. All of above labels are only for network meta-analysis, the label of “error symbol” represents that the software has not yet been specifically developed this feature up to now.
Software can also be divided into programming and non-programming categories, according to the programming functions. Software with only non-programming functions included GeMTC and ADDIS, while Stata and Microsoft Excel have both non-programming and programming functions. R contains the largest quantity of packages,17 with up to 11.
3.2 Involved methods
3.2.1 Theory summary
The basic theories of software performance are mainly related to the Bayesian theory21 and traditional frequentist22, 23 statistical theory. Compared with traditional frequentist statistics, the former has prominent advantages on a priori sets, risk-benefit ranking, and decision analysis, in addition to a more reasonable explanation of the confidence intervals (CIs) of the results, which is essential for the accuracy and integrity of the NMA results. Therefore, the calculation results based on the Bayesian theory have been highly respected for NMA.10
Usually, different a priori sets in Bayesian statistics directly lead to different results. Thus, the rationality of the a priori set is vital in performing NMA and is also one of the problems that causes concern for clinicians and statisticians.24 The software listed in Table 1, which are based on the Bayesian theoretical framework, call outside software or use their own functions when performing NMA.
Traditional frequentist statistical theory is applied only to R,25 Stata,26 and SAS27 software. In general, BUGS software is the most frequently used NMA software and is currently recommended by the Cochrane collaboration network.10
3.3 Model and implementation method
Currently, the NMA method is mainly based on the hierarchical model, regression model, multivariate analysis model, and the two-stage method. Relevant information is shown in Table 1.
The software based on the hierarchical model mainly call outside software based on the Bayesian theoretical framework or themselves to perform NMA. Stata software28 needs to call BUGS to perform NMA. Both proc genmod29 and proc glimmix30 from SAS software, which are based on the software programming function, use the traditional linear model and generalized linear mixed model to establish the hierarchical model; however, for performing NMA, the proc MCMC is based on the Bayesian theory.31
BUGS,32, 33 GeMTC/ADDIS,34 JAGS,35 Stan,14 and Microsoft Excel36, 37 use software based on a regression model based on the Bayesian theory. Microsoft Excel software can also achieve this by calling other software. R, Stata, and SAS software use the Bayesian theory and traditional frequentist statistical theory to perform NMA themselves or by calling other software. The BUGS software is the main called software, but the parts of the calling functions performed by the called software are different.
The software based on regression models include Stata,38 R,39 SAS,31 and Microsoft Excel.36, 37 The Stata software has the metareg module. R software has the rma and nlme packages. None of these can be used directly to perform multi-arm research; they must be transformed into another format for the study of both arms. The mixed process of SAS software is based on the design of the mixing-effect model; hence, it can be perfectly inserted into the regression model, the multivariate analysis model, and the two-stage method.
R software, which involves 11 types of packages32, 39-47 for NMA, is the programming software with the most widely used method, is the most flexible, and has the most complete set of functions. These packages mostly call outside software based on the Bayesian theoretical framework to accomplish NMA. Among these packages, all are required to establish suitable codes to achieve NMA, except the gemtc package40, 45 based on the self-hierarchical model. The nlme39, 44 and netmeta41, 43 packages are based on traditional frequentist statistics. The nlme package39, 44 uses the traditional linear model to achieve NMA but is not suitable for multi-arm research, and must be converted into another format to assess multi-arm research. At present, the netmeta package,41, 43 which is based on the electrical-network model theory to simulate the network data structure for analysis, is considered to be among the more mature packages, which are based on the theory of traditional statistics.
3.4 Software functions
3.4.1 Calculation function
Computing power is one of the most basic elements of statistical software. With the continuous updating of NMA methodology, which is based on the self-characteristics of the software and the differences in the implantation of the new methodology, the currently available software each have their own features.
Table 1 shows that each software has its own ability to independently implement NMA computing. Some software provided terminals where they could be called by external software. For example, BUGS software is the most popular software that is called. R, Stata, and SAS software, with their own independent operation abilities, not only are able to call external software but can also easily be called by outside software. Figure 1 displays the relationships between simulation software that are adopted for mutual co-operation forms to carry out NMA. In view of the different usages, the software for the usual NMA methods also has differences. For example, Stata software uses the metareg and mvmeta modules to accomplish self-calculation, calls outside software by WinBUGS, and is called by outside software such as R and SAS.
3.5 Graphing function
The relevance of the intrinsic structure of NMA data is more complex. This structure is often more difficult to use, as the performance is more limited if their own data are used to interpret their internal relations and reflect the results of the information; however, using graphics is better for reflection. Hence, software's ability to draw high-quality graphics has become one of the indicators with which to evaluate the software's quality, functionality, and operability. For NMA, one often needs to draw the following four types of graphics: network plot, forest plot, funnel plot, and ranking plot. The information contained in each of these four types of graphics is different, and their functions in NMA are also different.
The network plot,48 the main function of which is to reflect the intrinsic structure of the data, is one of the essential graphics in NMA. At present, the software that can draw it includes ADDIS, R, Microsoft Excel, and Stata. ADDIS and Microsoft Excel software, the graphics of which are generated automatically, are both non-programming software and can be easily used. The operation processes of R and Stata software are more complicated as their drawing functions are based on the programming code. However, their flexibility is better, as they can change the code program to draw a network plot that can reflect different information.
The forest plot,10 the most used graph in meta-analysis, not only can display the results well but can also simply visualize the corresponding heterogeneity. As shown in Table 1, five types of software can draw forest diagrams, the most frequently used of which are R and Stata software, which are associated with powerful drawing functions and flexible operability.
The funnel plot10 is used to detect the publication bias of original research. The detection of publication bias is a progressive study due to the current limitations of NMA methodology. Therefore, the use of graphics rendering is not common. Based on current methodology, the softwares equipped with the ability to draw these graphics are R and Stata software.
The ranking plot,49 originating from the idea of comparing the benefits and risks among various interventions, is a major feature of the NMA result graphics, and is drawn according to the risk-benefit probability of different kinds of interventions at each level. At present, the software that can draw the ranking plot include GeMTC, Microsoft Excel, ADDIS, R, and Stata software, of which the former three automatically generate the rank plot according to the corresponding figures, while the latter two use their own programming codes to achieve graphics rendering.
Only R and Stata are equipped with the ability to render the above four graphics at the same time, and both use special codes for drawing. Table 2 shows that all packages of R software perform NMA by themselves, but only some packages from R can draw the forest plot and probability plot. Rendering the other two graphics requires the use of other special program packages, for example, network packages to draw the network plot.
| Packages | Software(s) that are Called | Statistical Theory | Model | Forest Plot |
|---|---|---|---|---|
| R2WinBUGS | WinBUGS | Bayesian | All | Yes |
| R2OpenBUGS | OpenBUGS | Bayesian | All | Yes |
| BRugs | OpenBUGS | Bayesian | All | No |
| rbugs | OpenBUGS | Bayesian | All | No |
| R2jags | JAGS | Bayesian | All | Yes |
| rjags | JAGS | Bayesian | All | No |
| runjags | JAGS | Bayesian | All | No |
| rstan | Stan | Bayesian | All | Yes |
| gemtc | WinBUGS, OpenBUGS, and JAGS | Bayesian | Hierarchy model | Yes |
| nlme | / | Frequentist | Regression model | No |
| netmeta | / | Frequentist | other | No |
3.6 Comparison of calculation results
When software for performing NMA was selected, the accuracy of the calculative results was an important standard for selecting the software. The readability of the results and the convenience of extraction are also key factors in the choice of software. Table 3 shows an example of two nodes1, 2 (1, placebo; 2, bupropion) and the information about the results of the summary of the previous series of the article. We found that the results of each software were similar. When the Stata software uses the function by calling WinBUGS software, the generated results need to be collected by the external software to calculate the corresponding node's CIs, and the R software is usually used. The final results that are generated by three ways of calling the gemtc packages of R software are limited by the logarithmic effect, then, it is required for further operation for transition. The nlme packages of R software can generate the corresponding effect and standard error, but the result's CIs still needed to be calculated by the operators. By contrast, other software was more convenient for extracting the results, and the results based on Bayesian theory are more reasonable than those based on the traditional frequency theory in the interpretation of the CIs.10
| Software | Network Meta-Analysis Result Based on Itself | Software(s) with Function of Call | Package/Macros | Network Meta-Analysis Result Based on Packages/Macros |
|---|---|---|---|---|
| WinBUGS | 1.816 (1.310, 2.466) | Microsoft Excel | BugsXLA | 1.820 (1.315, 2.478) |
| Stata | WinBUGSfromstata | 1.818 (1.312, 2.473) | ||
| R | R2WinBUGS | 1.837 (1.318, 2.439) | ||
| gemtc | 1.794 (1.480, 2.107) | |||
| OpenBUGS | 1.826 (1.316, 2.471) | R | R2OpenBUGS | 1.826 (1.316, 2.471) |
| BRugs | 1.825 (1.315, 2.475) | |||
| rbugs | 1.804 (1.303, 2.470) | |||
| gemtc | 1.794 (1.480, 2.107) | |||
| JAGS | 1.820 (1.310, 2.456) | R | R2jags | 1.829 (1.317, 2.480) |
| rjags | 1.830 (1.313, 2.495) | |||
| runjags | 1.814 (1.281, 2.409) | |||
| gemtc | 1.798 (1.480, 2.116) | |||
| R | NA | R | nlme | 1.793 (1.528, 2.058) |
| NA | netmeta | 1.727 (1.380, 2.162) | ||
| GeMTC | 1.81 (1.32, 2.49) | NA | NA | NA |
| ADDIS | 1.80 (1.30, 2.48) | NA | NA | NA |
| Stata | 1.784 (1.313, 2.422) | NA | NA | NA |
| SAS (proc MCMC) | 1.737 (1.608, 2.092) | NA | NA | NA |
- Note: In addition to the results were automatic reserve 2 decimal in the GeMTC and ADDIS, other software were artificially 4 decimal. NA: not applicable.
4 DISCUSSION
The development history of NMA is longer than 10 years. Many difficulties due to methodology have been overcome to different degrees, followed by much research and development of corresponding software, and each software has its own characteristics.50 Software and its packages must continue to be developed and updated with the continuous maturity and perfection of methodology. In this article, a systematic summary of current software with functions for NMA is provided.
Based on the current NMA methodological defects and the operator's characteristics, the effectiveness of NMA is still being questioned.51 The support of superior methodology51 is the key to performing high-quality NMA, which guarantees the reliability of results and quality-grade evaluation of evidence, and also meets the needs of the comprehensive evaluation of the efficacy and safety of many intervention measures from clinical practice. Due to the theory, computer capability, available resources for its development, and comprehensive differences between software developers, different software has individual differences. Factors such as simple software operation, complete functions, and payable fees will determine whether the software will become popular or not. These are also the main problems that concern most software users. The openness of the software will directly determine the rate of usage. Compared with proprietary software, free software with a large number of complete functions represents the current mainstream software.
The research and development efforts and the speed at which the software is updated are driven by demand. The speed of the integration of the latest methodology is the key factor in maintaining the vitality. Compared with non-programming software, programming software has better openness, high speed of responses and updates, and flexibility of usage, but it is more complex. The choice of programming software mostly depends on the programming foundation and the habits of users. When performing NMA, programming software appears to be more flexible and is also updated faster. In the integration of the latest methodology, interfacing with external software, free user operation, and other aspects, programming software appears to be better. As is shown in Table 1, BUGS, JAGS, Stan software, and others are all based on the Bayesian theory. Stata and SAS software are mainly based on their own programming functions or calling other software to accomplish NMA, while R software is the most flexible. Only GeMTC and ADDIS are non-programming software, but they are updated more slowly. In addition, ADDIS software is dependent on GeMTC to accomplish NMA. The frequent updating of non-programming software will not only consume a large amount of manpower and material resources but also frequently affect the habits of the operators. To solve the problem of slow updating in a timely manner, non-programming software often specially develops functional interfaces with programming capabilities or calls outside software. Thus, such software is comparable to programming software in terms of renewal speed and the function-expansion aspect. However, non-programming software with simple operation is suitable for users with poor programming foundation and beginners.
From the perspective of methodology, we suggest that users choose software based on the Bayesian framework, the results of which are more reliable and the sorting function is better.4, 10 From the point of view of models, the hierarchical model, regression model, and multivariate analysis model can achieve NMA based on the Bayesian framework.52, 53 The two-stage method54 is often used in the frequentist framework. A variety of models have their own advantages and disadvantages, and the current controversy is relatively large. The key to choosing a model for the users is to know about the various model defects. We recommend that users follow the data structure and subsequent related results, including the heterogeneity test,55 consistency test,56, 57 and risk ranking.58
From the perspective of functions, good statistical software mainly depends on its data-processing and graphics-rendering abilities. Computational accuracy is directly related to the reliability of results, which is mainly related to the methodology contained in the software itself. It plays an important role in graphics display in NMA with great, comprehensive, concise, and easily understandable presentation. However, most software is focused on computing rather than on providing or partly providing the drawing functions. This should be a key point for future software development.
When choosing NMA software, we suggest that users comprehensively consider the following three aspects: (1) Preferred software should have both calculation and drawing functions, such as R and Stata software, based on the programming foundation and the habits of the users themselves. (2) Combine two or three types of software to achieve NMA. The combination of BUGS and R software is the preferred choice. The advantage is that the BUGS software has accurate and powerful data-processing ability based on the Bayesian framework and can also be perfectly combined with the drawing function of R software (or Stata software). (3) Operators accustomed to using non-programming software can choose GeMTC or ADDIS software. When necessary, the NetmetaXL macro may be chosen.
In conclusion, developing software as a statistical tool that is easy to operate, with complete functions, free (or inexpensive) access, and other characteristics, should be the direction of research and development. Due to the NMA's characteristics of complex production process and cumbersome operation, and the combination of multiple software programs, large powerful calculations, and the high performance demands of NMA, software with complete drawing ability, flexible operation, and user-friendly interface was researched and its development is urgent.
CONFLICT OF INTEREST
None.
