An Ontology to Model the International Rules for Multiple Primary Malignant Tumours in Cancer Registration

Round 1

Reviewer 1 Report

Authors are encouraged to clearly describe the motivation and significance of work in the introduction. Also, highlight the main contribution of the results.

The major issue with this work is lack of detailed evaluation. At least domain experts should be involved in the validation. How can one
validate the completeness of knowledge?

Lastly, authors are encouraged to include more related work. Also, consider validation (and even verification  ) of the rules. you may refer following papers

• Hussain, Maqbool, et al. "Acquiring guideline-enabled data driven clinical knowledge model using formally verified refined knowledge acquisition method." Computer Methods and Programs in Biomedicine 197 (2020): 105701.
• Afzal, Muhammad, Maqbool Hussain, Wajahat Ali Khan, Taqdir Ali, Sungyoung Lee, Eui-Nam Huh, Hafiz Farooq Ahmad et al. "Comprehensible knowledge model creation for cancer treatment decision making." Computers in biology and medicine 82 (2017): 119-129.

Author Response

The authors wish to thank the reviewer for the time and effort spent in reviewing the original article and for the comments to help improve its quality.

Please see the below for the post-by-point response.

----

Point 1: Authors are encouraged to clearly describe the motivation and significance of work in the introduction. Also, highlight the main contribution of the results.

Response 1: The introduction has been substantially revised to bring out the motivation for the work with regard to the devolution of the current centralised data-cleaning process of the European CR common data sets (described on lines 100-106). The main contribution of the results was described in the conclusions (lines 718-722) but an extra paragraph has been added (lines 743-748) concerning the potential deployment of the ontology and the message further emphasised in the additional text in the discussion section (lines 644-670). The work was essentially a proof of concept to show that even quite a complex set of rules – as characterised by those pertaining to multiple-primary tumours – can be modelled in description logic and applied to an ontology (lines 648-650). Since the multiple-primary tumour checks are the most complex of the various CR data checks that have to be performed, the result show that the whole data-cleaning software can be ontology-based.

Point 2: The major issue with this work is lack of detailed evaluation. At least domain experts should be involved in the validation. How can one
validate the completeness of knowledge?

Response 2: A number of the co-authors are in fact domain experts who contributed to the development of the ontology and checked the axioms. The aim of the work was not intended to validate the multiple-primary tumour rules themselves but to validate cancer-registry common data sets against those rules. Regarding verification (c.f. final comment), the examples provided in section 3 are not intended to be comprehensive of all possible combinations of topography and morphology codes, but rather to show each one of the different error-trapping scenarios that the axioms have to contend with (described in section 2.4). These mechanisms, which handle all the different possible scenarios, have been shown to behave appropriately on the basis of the underlying logic and there is no reason why they should behave differently for any other tumour-couplet combinations. As noted in the modified text however, the next stage will be to test the ontology comprehensively with the full CR test data set and thereafter with the full sets of CR data (lines 650-652, discussion section).

Point 3: Lastly, authors are encouraged to include more related work. Also, consider validation (and even verification  ) of the rules. you may refer following papers

• Hussain, Maqbool, et al. "Acquiring guideline-enabled data driven clinical knowledge model using formally verified refined knowledge acquisition method." Computer Methods and Programs in Biomedicine 197 (2020): 105701.
• Afzal, Muhammad, Maqbool Hussain, Wajahat Ali Khan, Taqdir Ali, Sungyoung Lee, Eui-Nam Huh, Hafiz Farooq Ahmad et al. "Comprehensible knowledge model creation for cancer treatment decision making." Computers in biology and medicine 82 (2017): 119-129.

Response 3: The authors are grateful to the Reviewer for the interesting references. An additional sub-section (entitled “Validation of health data” – section 1.1, lines 161-180) has been included in the introduction where these references have been added [20-21] along with further ones [17-19, 22] and mentioning their different aims. Further explanation has also been provided as to how the validation of CR common data sets has a somewhat different focus from these other initiatives (lines 184-194). Nevertheless, the references are of interest and could potentially be applied to automatically deriving/learning some of the other validation rules directly from the cancer-registry data sets.

Reviewer 2 Report

The authors present in this paper a whole modeling process of an ontology in order to represent in formal language (description logics) the international rules for multiple primary malignant tumors in the context of cancer registration. 

The paper is well written, and the study is very interesting. I have however several concerns. 

The first one is related to the lack of study/comparison with other existing ontologies and positioning in regards to the application domain. If there is prior work from others, a brief summary of this is needed in the paper. If not also. Also explain why existing ontologies do not fit with this application domain.
More detail is needed on the final application and its evaluation in terms of usability and I think it would be of great interest for the readers.

I don't think that all the detailed axioms are needed here.

The authors did not mention at all their positioning concerning rules in SWRL. In conceptualizing their ontology, would it be relevant to add rules ? If yes, please detail, if not please explain why.

I was wondering also why the authors did not mention temporal and/or spatio-temporal information and reasoning that adaptation to the context domain would be challenging but relevant.

The authors also mentioned briefly a DL-based querying . I think it would be interesting to go further into this direction, with some use cases that would help in validating the modeling process and the proposed axioms.

Also, how this ontology and those axioms will be used/shared with others except inside the dedicated application ? Is it planned to publish it in well-known portals (BioPortal, OBO Foundry ?) and mapped with other ontologies ? 

The output text explaining violation of the MPT rules for the tumor permutations and reasons provided is of a great interest but are not understandable (unless the end-used is very familiar with the outputs of the OWL-API). An evaluation is also needed here.  

Author Response

The authors wish to thank the reviewer for the time and effort spent in reviewing the original article and for the comments to help improve its quality.

Please see below for the post-by-point response.

----

Point 1: The paper is well written, and the study is very interesting. I have however several concerns. The first one is related to the lack of study/comparison with other existing ontologies and positioning in regards to the application domain. If there is prior work from others, a brief summary of this is needed in the paper. If not also. Also explain why existing ontologies do not fit with this application domain.

Response 1: The ontology reported here is essentially addressed to one of the cancer-registry data validation checks that was only handled in a rudimentary fashion in a previous ontology (c.f. introduction, lines 145-151). The latter was reported in an earlier publication [15] which discusses the ontology in relation to the other ontologies that have been referenced very briefly here (lines 140-144). As far as the authors are aware, there have been no other initiatives using ontologies directly for validation of CR common data sets and this has been explicitly stated in the modified text, lines 237-239 (section 1.3). A new section has also been added to the introduction entitled “Data validation using OWL” (section 1.2, lines 196-229) where it is argued that OWL is not necessarily the most appropriate tool for data validation owing to its open world assumption. Using OWL to validate data implies certain design constraints that are generally at odds with standard ontologies. Care also has to be taken regarding the number of axioms within the ontology in order not to compromise reasoning performance, which also limits the reuse of large standard ontologies. All these aspects have been addressed in the added text.

Point 2: More detail is needed on the final application and its evaluation in terms of usability and I think it would be of great interest for the readers.

Response 2: The discussion section has been amplified with additional text (lines 644-670, 702-716) that addresses the intended purpose of the MPT ontology. Whereas the ontology can function as a standalone application for testing multiple-primary tumour cases, the primary purpose is to integrate it into the ontologies dealing with other cancer-registry data checks. The aim once this ontology tool-kit has been thoroughly tested is to swap out the current quality-check software and move to a more federated approach regarding the cleaning of individual cancer-registry common data sets. The aims are explicitly stated in the updated text of the abstract (lines 19-23, 31-34) and within the updated main text (introduction: lines 97-106, 115-118, 145-151, 157-158, 184-194; discussion: 648-650 conclusions: 739-748).

Regarding usability, as noted in the additional text forming the last paragraph (lines 709-716) of the discussion section, the internal mechanisms of the ontology can be hidden by a front-end application to provide a user-friendly interface to running the data checks. This is essentially the situation with the current quality-check software that is freely downloadable. An ontology-based solution however offers a number of inherent advantages and also affords the user with an interactive and self-contained tool to explore and understand the data model. Whereas it is questionable if the majority of users will take/have the time to become initiated with OWL, Protégé offers a relatively user-friendly interface and – for the initiated at least – having access to the rule base in axiomatic form is likely to save substantial time. There is also no reason why the existing rule-based tables cannot be automatically generated from the ontologies and it would mean that only one repository of rules would have to be maintained. It will in any case be interesting to understand how the ontologies might be used within the cancer registries once they have been fully tested and deployed as a pilot initiative in some cancer registries.

Point 3: I don't think that all the detailed axioms are needed here.

Response 3: All the ABox axioms have been removed from the main text and placed in an Appendix (Appendix B) – we did not remove them altogether since they provide a reference for the descriptions of Figures 3-6. The TBox axioms describe the various error-trapping scenarios that form the basis of the results section and are referred to several times so these have been retained in the main text.

Point 4: The authors did not mention at all their positioning concerning rules in SWRL. In conceptualizing their ontology, would it be relevant to add rules ? If yes, please detail, if not please explain why.

Response 4: Additional text has been inserted in the discussion section to address SWRL (lines 702-708). SWRL extends the expressivity of OWL DLs and has a set of built-ins that are particularly useful for dealing with temporal expressions, but it comes at a potential cost of interoperability and decidability.  In addition, the FACT++ reasoner does not support SWRL and in order to avoid these sorts of restrictions and reduce maintenance, SWRL was not used; OWL was able to model all the requirements.

Point 5: I was wondering also why the authors did not mention temporal and/or spatio-temporal information and reasoning that adaptation to the context domain would be challenging but relevant.

Response 5: The multiple primary rules do not include the need to work with temporal or spatial information. Other of the cancer-registration validation checks do have to check chronology of certain dates but these checks can also be handled in the pre-processing code without the need for using SWRL for example.

Point 6: The authors also mentioned briefly a DL-based querying. I think it would be interesting to go further into this direction, with some use cases that would help in validating the modeling process and the proposed axioms.

Response 6: A new section entitled “DL query interface” has been added (section 3.2, lines 557-641) together with 7 new figures (Figures 11-17) to address a number of use cases that can be used to aid the modelling process. The DL queries and reasoner-provided explanations have been cross-referenced with other figures and relevant axioms to help further explain the violation of the MPT rules (with reference to the Reviewer’s final comment).

Point 7: Also, how this ontology and those axioms will be used/shared with others except inside the dedicated application ? Is it planned to publish it in well-known portals (BioPortal, OBO Foundry ?) and mapped with other ontologies ? 

Response 7: The MPT ontology will form part of a suite of ontologies with the aim eventually of swapping out the current quality-check software for European cancer registries. Although the ontologies are specific for cancer-registry data validation, they do build on base ontologies that could very usefully be standardised – such as the ICD-O-3 codes. Entities within the ontologies could also be annotated with references to similar entities in current standard ontologies such as the NCIt cancer thesaurus. This is planned once the ontologies have been thoroughly tested in the field.

Point 8: The output text explaining violation of the MPT rules for the tumor permutations and reasons provided is of a great interest but are not understandable (unless the end-used is very familiar with the outputs of the OWL-API). An evaluation is also needed here.  

Response 8: The examples provided in the new section 3.2 (“DL query interface”, lines 557-641) and the elaboration of the associated reasoner-provided explanations should hopefully provide further clarification of the outputs from the OWL-API. The outputs of the OWL-API are essentially the same as those illustrated in the figures taken from the Protégé user interface and trap the same errors. The examples of the DL query interface should further bring this out. The examples included in the paper tackle all the different types of error-trapping scenarios that have to be handled and should not behave differently for different combinations of topography/morphology codes. As described in the modified discussion text however, the next stage will be to test the ontology comprehensively with the full CR test data set and thereafter with the full sets of CR data (lines 650-652).

Round 2

Reviewer 1 Report

The authors have addressed most of my comments. However, it would have been great if significance of approach could be generalized. The added text during revision require minor revisions to eradicate MINOR grammatical issues.