Zobel’s Checklist

Since I don’t have a master thesis topic yet, I used the paper:

Paul, C., et.al.: Efficient Graph-Based Document Similarity. In:
Proceedings of the ESWC 2016, http://doi.org/10.1007/978-3-319-34129-3_21

Regarding hypotheses and questions,

  • What phenomena or properties are being investigated? Why are they of interest?
    • The paper investigates automated document similarity meassurement. This is for example used in article recomendations on newspaper websites.
  • Has the aim of the research been articulated? What are the specific hypotheses and research questions? Are these elements convincingly connected to each other?
    • The aim is to present a graph-based algorithm to measure semantic similarity of documents, that:
      • (i) provides higher correlation with human notion of similarity than similar approaches
      • (ii) first hypothesis also holds for small documents with few annotations
      • (iii) is more efficient than other graph based approaches
  • To what extent is the work innovative? Is this reflected in the claims?
    • The algorithm is said to be more efficient than other graph-based algorithms. The used similarity measure is said to correlate more with human notion than comparable ones.
  • What would disprove the hypothesis? Does it have any improbable consequences?
    • (i) and (ii) similar approaches provide equal or higher correlation with human notion
    • (iii) An existing graph-based algorithm that is equally or more efficient with similar or better results
  • What are the underlying assumptions? Are they sensible?
    • Semantically annotated documents as are available as input. This is sensible because it helps to focus on comparing the documents rather than analyzing them.
  • Has the work been critically questioned? Have you satisfied yourself that it is sound science?
    • The paper shows that the proposed algorithm is better than some selected other ones on selected data. As I am not into the topic of semantic document comparison I cannot say whether the selected data and reference algorithms are representative. Additionally, the paper never states the limitations of the proposed algorithm. So the results do not appear very trustworthy to me.

Regarding evidence and measurement,

  • What forms of evidence are to be used? If it is a model or a simulation, what demonstrates that the results have practical validity?
    • An experiment on 2 datasets (one standard benchmark set and another with small documents)
  • How is the evidence to be measured? Are the chosen methods of measurement objective, appropriate, and reasonable?
    • The authors use standard metrics that were also used to evaluate the reference algorithms. So the 3 criteria seem to be fulfilled.
  • What are the qualitative aims, and what makes the quantitative measures you have chosen appropriate to those aims?
    • The authors want to show that their proposed algorithm comes closer to human notion of similarity and works more efficient than reference algorithms. Using the standard metrics is appropriate for that.
  • What compromises or simplifications are inherent in your choice of measure?
    • I am not well familliar with the used meassures, so I cannot say anything here.
  • Will the outcomes be predictive?
    • Yes, the hypotheses predict higher similarity meassures and less excecution time for the proposed algorithm in comparison to similar ones.
  • What is the argument that will link the evidence to the hypothesis?
    • The quantitative measures allow to directly compare the performance of the proposed algorithm with reference algorithms. (Is it faster? Is it closer to human notion?)
  • To what extent will positive results persuasively confirm the hypothesis? Will negative results disprove it?
    • Since the authors do not state any constraints, they indirectly claim their algorithm performs better than reference ones in any circumstances. There for a positive result in an experiment may strongly support their hypothesis but not totally confirm it. However, negative results will directly disprove their hypothesis.
  • What are the likely weaknesses of or limitations to your approach?
    • I could not find any statements of the authors regarding weaknesses or limitations of their proposed approach

One thought on “Zobel’s Checklist

  1. Sebastian says:

    Hey Manuel,

    If I remember it rightly, some students said that it is difficult to understand the text. For me as a non-expert in computer science it is also even more worse to understand. After I read the homework of all students, I’ve got a comprehensive overview of the content of the text. But I can’t know and understand every detail.
    My impression about your homework and solution is that you are on a really good way. You understand the core of and all your answers have the same content like the other homework I had read. Concerning the Question ”How is the evidence to be measured? Are the chosen methods of measurement objective, appropriate, and reasonable?” I noticed that your answer is correct. But here I can add the fact, that the authors will fail to state which hardware they used in their experiments. This seems like an important information to me. You also not sure about the question “What compromises or simplifications are inherent in your choice of measure?“ That is no problem, I think you can take a look at the explanation of the other students.
    But all in all your homework and explanations look really good. Well done.

Leave a Reply

Your email address will not be published. Required fields are marked *