Overview of the European post-authorisation study register post-authorization studies performed in Europe from September 2010 to December 2018

1 INTRODUCTION

The European post-authorisation study (EU PAS) register is a publicly available repository of post-authorisation (PAS) studies developed and supported by the European Medicines Agency (EMA) through the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP).¹ Current pharmacovigilance legislation in Europe requires to make the study protocols and summary of results publicly available for post-authorisation safety studies (PASS) imposed as a condition to the Marketing Authorisation (RMP category 1) or as a specific obligation in the context of a Marketing Authorisation under exceptional circumstances (RMP category 2). This register is also aimed to host non-imposed studies, such as those required as per Risk Management Plan (RMP category 3)² and all observational studies performed on authorised medicinal products, including effectiveness studies. The EU PAS register is the platform through which such studies, as well as other non-imposed studies, are made accessible online. Indeed, the aim of the EU PAS register, in line with ENCePP's mandate, is to improve the transparency of studies conducted within the EU area and beyond.

The EU PAS register has been described in detail in several previous studies. Engel & Almas et al, in a review of 189 PASS identified in the Pharmacovigilance Risk Assessment Committee (PRAC) Meeting Minutes from July 2012 to July 2015,³ searched the availability of those studies in the EU PAS and showed that only 93 of the studies were registered in the EU PAS register and among them only 43% had available protocols. Analysis of PRAC comments identified several methodological concerns, and limited reasoning behind the methodological decisions and feasibility considerations. Slightly more PASS used primary data collection (i.e., data prospectively collected for the purpose of the study) except those studies assessing drug utilisation where secondary use of already existing healthcare data was leveraged. Another review, conducted by Carroll et al and focusing on studies available on the EU PAS register as of October 2016,⁴ corroborated that primary data collection was more common in studies aimed at assessing safety and effectiveness and less common when assessing drug utilisation. In 2018, Vora et al and Farcas et al,^{5, 6} used the EU PAS register to explore specifically studies evaluating the implementation and impact of risk minimisation measures.

All these previously conducted EU PAS register-based studies relied primary on the data reported within the register, without validation of the collected information or additional information on methodological aspects based on expert review. Another gap in these previous studies is the lack of focus on multiple-database studies (MDS), which are observational studies conducted using more than one source of routinely collected data (e.g., claims database, electronic health records [EHRs]). MDS are of particular importance because they allow the accrual of a very large cohort of patients, which is of particular relevance to paediatric populations and rare diseases, as well as several other populations of special interest. Indeed, the number of MDS has been increasing over the years. Since EU PAS register is a platform for the mandatory recording of data on observational studies as per EU legislation, identifying and describing such studies and their impact on the landscape of observational research is therefore of great value as this has never been done to our knowledge.⁷

Given how quickly observational research is growing, the aim of the present descriptive study was to conduct an updated and detailed review of the studies that were registered in the EU PAS register from its inception till the end of 2018, providing additional information on data type (e.g., distinguishing between claims data, EHRs, etc.) and study design (e.g., distinguishing between descriptive studies, cohort studies, case–control studies, etc.). Another aim of this study was to conduct an assessment of the inter-rater reliability following the collection of data.

2 METHODS

2.1 Data collection

A dataset containing all studies found in the EU PAS register from its inception to 31st December 2018 was provided by the EMA. The EU PAS register is publicly accessible online (http://www.encepp.eu/encepp/studySearch.htm). The data collection was carried out independently by 34 investigators from 13 academic centres or contract research organisations being part of ENCePP on common and detailed instructions and using the same electronic case report form (Figure 1). The resulting dataset contained data from the EU PAS register concerning different aspects of study transparency (ENCePP Seal, protocol and availability of publication), regulatory obligations, methodology, target population, scope and drug under study (chemically synthetised vs. biological drugs, orphan drugs). EU PAS register data was supplemented by retrieving data from study protocols or publications, if available, on source of funding, whether a study was an MDS (defined as a study using more than 1 source of already existing databases which could not be linked at patient level), study design, use of reference drug for formal comparison (if any). The full protocol for data collection, including the fields provided by the EMA, is provided in Appendix S1A. Once all the data was collected it was harmonised based on pre-defined criteria.

To evaluate how consistent data collection was, data retrieved by investigators from a centre was checked by investigators from another centre for a 15% random sample of studies using the same protocol used for the main data collection; each investigator validated five studies. Any disagreements were resolved through the intervention of a third centre. This was done for the following key variables: setting, study type (new classification), study design, type of secondary data used, whether the study was an MDS, use of reference drug for formal comparison, drug type, whether the study drug was an orphan drug. After data collection was completed, automated quality check of data entry was conducted for the following fields: study type, data collection method, type of secondary data if applicable, whether the study was MDS, study type and study design. In brief, all investigators were asked to collect de novo data concerning these fields while blinded to previous assessment done by investigators belonging to a different centre.

3 RESULTS

After excluding two duplicates, a total of 1426 PASs were identified in the EU PAS register from its inception to 31st December 2018. The majority of studies were observational (N = 1227; 86.0%). Almost a quarter of all observational studies were based on primary data collection (N = 299; 24.4%; Figure 2). Two-thirds of observational studies were based only on secondary use of existing healthcare data, that is, data not collected primarily for research purposes (N = 757; 61.7%). N = 299 (24.4%) were based on primary data and N = 147 (12.0%) were based on both primary data collection and secondary use of databases.

The frequency of studies registered in the EU PAS register showed a steady increase in the number of observational studies from 2010 to 2016, while a slight decrease was observed during the last 2 years under study (Figure 3). Almost a third of observational studies (N = 718; 58.5%) had a protocol deposited in the EU PAS register (Table 1), similar to most study types except for clinical trials, where only N = 11 (36.7%) had a protocol deposited. The protocol was available for 63% (N = 360) of observational studies requested by a competent authority. Only 6.9% of all observational studies had information on ENCePP Seal, while the availability of a publication was much more common: about a third of all observational studies had a publication available. Just under half of all studies were required by a competent authority. The most common type of risk management plan (RMP) classification for observational studies was RMP 3, that is, required by a competent authority (29.3%). Information on the RMP status was missing in 53 (4.3%) observational studies and it was classified as ‘non-EU RMP only’ for 102 (8.3%) of them.

The scope of most observational studies was risk assessment (56.7%), followed by drug utilisation (36.2%). A large proportion of observational studies was conducted included the elderly (82.2%). Only 36.0% (N = 442) of observational studies were conducted included children, and not all of these studies included children exclusively. With regards to drugs of interest, biologic drugs were studied in 24.3% of all observational studies; orphan drugs were the focus of 12.8% of observational studies.

Concerning the type of secondary data used, among observational studies the use of claims, EHRs or existing registries was similar, at 13.4, 14.9 and 11.1%, respectively (Figure 4). The most commonly used study design among observational studies was the cohort study (52.4%), followed by other types of descriptive studies (23.5%), such as drug utilisation studies, disease epidemiology and pharmacokinetics (Figure 5). Overall, N = 319 (26.0%) of observational studies were classified as MDS.

The overall percentage of agreement in data categorisation among collaborating centres that collected the data concerning study variables was highest for study type (93.5%) and lowest for type of secondary data (67.8%; Table 2). The low level of agreement for secondary data is expected to have an impact on the overall level of agreement. These results were largely in line with total kappa coefficients. The values of Cohen's kappa and the centre variations in Cohen's kappa for key, along with their 95% CIs, are shown in Figures S1B1 and S1B2, respectively.

Compared to studies based on the secondary use of data, those based on primary data collection were less likely to have a protocol deposited, to be funded by public entities and to use a reference drug for formal comparison, while they were more likely to be funded by pharmaceutical companies (Table 3). In general, there was no substantial difference in study design between studies based on primary data collection or secondary use of data, although descriptive studies were slightly more common in the former.

Only a third of all claims- and EHR-based observational studies were requested by a regulator (Table 4). Claims data were more commonly used for risk assessment than EHRs (60.8% vs. 40.7%).

Among observational studies, as compared to non-MDS, MDS were more likely to have a protocol deposited in the EU PAS register (OR: 1.9; 95% CI: 1.4–2.5), to have an ENCePP Seal (OR: 3.0; 1.9–4.7), to be funded by national/international drug agencies (OR: 1.9; 95% CI: 1.1–3.4) and to include children as study population (OR: 2.1; 95% CI: 1.5–2.6). Moreover, almost two-thirds of MDS were requested by a regulator. Interestingly, drug utilisation was significantly more common as a study scope in MDS compared to non-MDS studies (OR: 1.3; 95% CI: 1.1–1.7). However, MDS were less likely to consider orphan drugs as the main exposure (OR: 0.5; 95% CI: 0.3–0.8) (Table 5).

4 DISCUSSION

The present study provides a detailed description of all studies registered in the EU PAS register from its inception until the end of 2018, focusing and providing detail on various aspects of study design and multiple database studies specifically. As expected, as compared to the most recent review of EU PAS register available,⁴ our results showed a steady increase in the number of observational studies registered. The present study adds to the available literature by providing the most recent updated review of the EU PAS register, including all PAS rather than just PASS. To do this, we leveraging a network of pharmacoepidemiologists from various centres belonging to ENCePP to collect additional and detailed information on methodological aspects of observational studies. Furthermore, to our knowledge, the present study is the only one that validates the inter-rater reliability of different collaborating centres during the data collection process, thus providing indirect measure of the completeness and accuracy of the data collected in the EU PAS register.

From our study, it appears that only 61.7% of observational studies were conducted through the secondary use of existing healthcare data. This finding indicates that the remaining one third of observational studies were conducted using the much more resource-intensive method of collecting data ad hoc, that is, primary data collection. It is not known whether such studies could have been easily conducted using routinely collected healthcare data, which provide detailed information on population characteristics, exposure and outcome. In general, we expected fewer studies to have been conducted using primary data, as large use of primary data source can reflect the need to improve the access to data for research purpose. In line with previous studies, primary data were more commonly used for effectiveness and risk assessment studies,^{3, 4} while secondary data was more likely to be used for drug utilisation studies and risk assessment studies. The increase in use secondary data source was sharpest in the last 2 years, while until 2016 the use of primary data source was in in 56% of the 316 studies identified in the EU PAS register.⁴

Of note, some markers of high ethical standards and transparency such as protocol deposition are often missing. This result has to be balanced with the fact that registration is legally binding and subject to financial penalties only for imposed PASS at the time of final study report. Finalised studies that are RMP 1 and 2 with protocol available were 68.5%. It was found that 37% of clinical trials did not have a deposited protocol while 58% of observational trials did. Of 1204 observational studies, 15 (1.2%) had no information whatsoever on study design, highlighting the importance of transparency. Transparency in research along with methodological rigour are essential to increase the standards of pharmacoepidemiological research in Europe.⁹ Competent authorities should keep emphasising the importance of study registration in order to increase the level of transparency in the research landscape, particularly for publically funded research. Another measure of high ethical standards and transparency, the ENCePP Seal, was only available in 7% of all observation studies. The ENCePP Seal identifies studies in the register that adhere to the entirety of the Code's provisions, including timelines for the submission of documents to the ENCePP Secretariat. However, obtaining this Seal is associated with an added bureaucratic burden that researchers may not be willing to undertake. The usefulness of the ENCePP Seal on a broader scale may be somewhat limited if so few studies undertake the procedures to obtain it. Other initiatives such as the revision of the ENCePP Code of Conduct have recently been undertaken but it is not known to what extent this is being adopted.¹⁰

Another interesting finding of this study with important implications concerns the level of agreement seen between pharmacoepidemiologists in the process of data collection from the EU PAS register. While there was substantial agreement for study type, data collection, drug type and use of reference drug for formal comparison, there was only moderate agreement for the rest of the fields, including study setting, classification of secondary data, whether the study was an MDS and whether the study drug was an orphan drug. High agreement was not reached for any of the fields, due to ambiguous, missing or conflicting reporting. These findings are thought provoking and suggest that data in the EU PAS register was not always clear and complete. Indeed, the quality of data entry in the EU PAS register is not monitored at all as with other similar platforms such as clinicaltrials.gov. In the latter, National Library of Medicine (NLM) staff conducts a review of registered study records for obvious errors or inconsistencies, with important issues being communicated directly to the investigators.¹¹ However, this does not guarantee complete and accurate records for all studies.¹² In the EU PAS register, it is completely up to the investigator's discretion to conduct data entry correctly and accurately as no quality control is conducted. The very small number of duplicate studies identified, two, is potentially an indicator that in this sense, the register's data is of satisfactory quality. In addition, there are several open text fields, missing categories in the different attributes and certain categories used in more than one attribute which makes it challenging to identify and summarise study types in the EU PAS register information. It should be noted that there is currently no glossary in the EU PAS register to promote harmonisation of terms used. We argue that the terminology to be used should be based on a single accepted definition that is presented in a glossary. The reason is that the classification of studies and other relevant aspects (e.g. data sources, methods etc.) was often found to be conducted using ambiguous or even incorrect terms. For the EU PAS register to hold promise of a greater transparency and accuracy, by serving as a common repository to share pharmacoepidemiological research, it would be key to further standardise its data elements, and create completion guidelines to offer an unequivocal interpretation of the fields by study registrants.

With our working definition of MDS as studies which use more than one secondary data source, we found more than 20% of observational studies classified as MDS. These studies were more likely to have the ENCePP Seal and to have protocol deposited, all markers of high transparency. Orphan drugs were not commonly investigated using MDS, although such studies may have a lot of potential in the rare disease field given the small population sizes expected and the increasing need of merging data from different sources to speed up the development of these medicines.

Our analysis has several strengths compared with previous studies as it provides a detailed overview of the studies registered on EU PAS register, with systemic data collection being conducted by a network of established centres of excellence in pharmacoepidemiology. It is very important to underline that data collection was carried out by researchers independently based on common and very detailed instructions. These instructions were shared, agreed and made accessible and available among all reviewers, to avoid potential errors in the classification method. Again, to further check, blind quality control of data entry was conducted for some fields, to obtain an overall agreement. In addition, this is the first time that an evaluation of the MDS landscape has been attempted.

However, this study also has some limitations. The most important limitation concerns the accuracy of the data collection, which was sometimes limited by the lack of information recorded in the EU PAS register or lack of clarity of such information. The lack of data and ambiguities in classification of some studies may limit the robustness of our analyses. In these instances, the judgement on categorising characteristics of a study was somewhat subjective. This is a limitation inherent to the way that data is collected in the EU PAS register and to the lack of quality control of such data. Furthermore, since the registration of a study is voluntary, the EU PAS register is not representative of the whole landscape of studies in Europe and some studies may have not been captured. Another limitation is that the data was collected only until 2018. However, we hold that the present study still has the added value of providing an overview of the EU PAS register compared to the most recently published reviews, which described studies until 2016.⁴

5 CONCLUSION

Observational studies were the most common type of studies in the EU PAS register, but almost one third of observational studies used primary data, which is more resource-intensive. One quarter of observational studies were MDS. MDS hold untapped potential to investigate special populations, such as patients with rare diseases and paediatric patients. A detailed analysis of IRR during data collection suggested that information recorded in the EU PAS register is often not clear or complete. Increasing the quality of study-related information would considerably increase the transparency of research.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

ETHICS STATEMENT

For this study, there was no need to set up or directly manage databases in which there might have been identifiable or identified patient data; therefore, according to local regulations, obtaining an ethics committee approval to carry out this study was not necessary.