A Framework for the Evaluation of SUMO-Based Ontologies Using WordNet

This paper offers a new practical approach toward automated commonsense reasoning with first-order logic (FOL) SUMO-based ontologies. We propose a new black-box evaluation framework for SUMO-based ontologies, which exploits the world knowledge encoded in WordNet and its mapping into SUMO. Our proposal consists of both a novel semi-automatic method for the creation of a large set of commonsense problems and a new procedure that enables its automatic evaluation by using automated theorem provers (ATPs). The application of our method enables the creation of a very large benchmark consisting of more than 15 000 problems from a small set of manually built question patterns that exploit the WordNet semantic relations. By means of the resulting benchmark, we successfully evaluate the competency of different translations of SUMO into FOL and the performance of various state-of-the-art FOL ATPs according to several quality criteria. A general analysis of our experimental results demonstrates that the proposed commonsense problems are heterogeneous and non-trivial. Furthermore, a fine-grained analysis of the experimental results obtained for a sample of our benchmark enables the detection of some mapping errors and some discrepancies between the knowledge of WordNet and SUMO. The evaluation benchmark and all the resources that have been used and developed during this work are released in a single package.