This video shows a web application that we have developed and used for building a gold standard for tuning and evaluating Tìpalo algorithm. The resulting resource can be downloaded at http://stlab.istc.cnr.it/documents/iswc2012/gold_standard.csv The application manages argumentation between users in order to support them in reaching agreement.

The schreencast shows:

• a user logging into the application and performing the required task: given an entity and its description, the task is to indicate the most appropriate types for that entity by (i) copying and pasting the term that expresses the entity type in the definition sentence (this is used as an indicator that the user has clearly understood the task, and we can use this value also for comparing with the terms we select automatically), (ii) select the most appropriate types from two different lists of ontology types (for each type the tool provides a definition and some examples);
• a second user logging into the application and performing the same task as above. However, such user is provided with an entity that he/she had previously analysed. This means that other users performed the task on such entity and that there is disagreement among them (< 70%). In this case the user can visualize other users input and decide to either change his/her values or keep it as it is. In either cases he/she has to provide an explicit motivation that can be used for argumentation purposes.

This video shows a web application that we have developed and used in order to evaluate the quality of Tìpalo results. The resulting resource (including users judgements) can be downloaded at http://stlab.istc.cnr.it/documents/iswc2012/user_evaluation.csv The screencast shows a user logging into the application and performing the required evaluation task: given an entity and its definition, users are asked to assess the correctness/soundness of:

• the types automatically assigned to such entity;
• the identified taxonomies for such types;
• the identified meaning for such types.

Users express their judgment based on a three-value scale (no, maybe, yes).