==+== Paper Reproducibility Review Form ==+== For more information, refer to: ==+== https://www.acm.org/publications/policies/artifact-review-badging and ==+== https://www.sigcomm.org/content/sigcomm-artifacts-evaluation-committee ==+== ===================================================================== ==+== Begin Review ==+== Reviewer: (Your name here) ==+== Paper Name: (Paper name here) ==+== A. Type of Artefact ==-== Choices: ==-== 1. Dataset ==-== 2. Script/Tools ==-== 3. New Artefact ==-== 4. Extension of Existing Artefact ==-== Enter the number of your choice: (Your choice here) ==+== B. What is the degree of efforts to obtain the artefact? ==-== Rate from 0% to 100%, where 0% means NOT AVAILABLE; 20%, VERY HARD ==-== TO OBTAIN up to 100% meaning VERY EASY TO OBTAIN. ==-== Choices: ==-== 1. 0% ==-== 2. 20% ==-== 3. 40% ==-== 4. 60% ==-== 5. 80% ==-== 6. 100% ==-== Enter the number of your choice: (Your choice here) ==+== C. Comment about the degree of efforts to obtain the artefact ==-== E.g., did the paper have explicit pointer to the artefact? The ==-== artefact is simply not available. I need to register to obtain the ==-== artefact. The artefact is a testbed and I cannot use it. The ==-== artefact is actually hardware, I cannot obtain it. The artefact is ==-== licensed and I should pay... ==+== D. Do you consider that the nature of the artefact (or any component ==+== of it) may raise (non-technical) issues? Of which Nature? ==-== E.g., ethical issues; non FRAND - Fair, Reasonable and Non ==-== Discriminatory - licenses; environmental issues; safety issues;... ==+== E. Has the artefact been deposited in what can be reasonably ==+== considered the most appropriate available repository, which ==+== ensures perennially availability? ==+== F. In relation to the previous point, how long do you expect or ==+== estimate the artefact will be available: ==-== Choices: ==-== 1. Less than 1 year ==-== 2. Between 2 and 5 years ==-== 3. Between 5 and 10 years ==-== 4. More than 10 years ==-== Enter the number of your choice: (Your choice here) ==+== G. Are the instructions on how to install and make the artefact ==+== functional provided? ==-== Rate from 0% to 100%, where 0% means NOT INSTRUCTION PROVIDED; 20% ==-== VERY HARD TO UNDERSTAND up to 100% meaning VERY EASY TO INSTALL ==-== AND RUN. ==-== In case of a data set the question has to be answered considering ==-== 0% means NOT DATA FORMAT DESCRIPTION; 20% VERY HARD TO UNDERSTAND ==-== THE DATA FORMAT up to 100% meaning VERY EASY TO UNDERSTAND AND USE ==-== THE DATA. ==-== Choices: ==-== 1. 0% ==-== 2. 20% ==-== 3. 40% ==-== 4. 60% ==-== 5. 80% ==-== 6. 100% ==-== Enter the number of your choice: (Your choice here) ==+== H. Comment about the efforts to obtain, install, and use the artefact ==-== E.g., very limited instructions were given; I abandoned at some ==-== point because it was not clear what to do; it was difficult to ==-== figure out what exactly are the steps to follow; easy to follow ==-== but some degree of familiarity is necessary; very simple ==-== step-by-step instructions have been provided; piece of cake, the ==-== authors provide a guide for dummies; I had nothing to do I just ==-== downloaded a VM image. ==+== I. If the artefact reached the "functional" level, meaning that it ==+== seems to function correctly, did the authors provide all of the ==+== information and meta-data needed for others to reuse the artefact? ==-== Rate from 0% to 100%, where 0% means NO DOCUMENTATION PROVIDED; ==-== 20% DOCUMENTATION HARD TO UNDERSTAND up to 100% meaning ==-== DOCUMENTATION VERY CLEAR AND EASY TO UNDERSTAND. ==-== Choices: ==-== 1. 0% ==-== 2. 20% ==-== 3. 40% ==-== 4. 60% ==-== 5. 80% ==-== 6. 100% ==-== Enter the number of your choice: (Your choice here) ==+== J. Comment about the quality of the documentation ==-== E.g., documentation is complete and clear; documentation is ==-== lacking fundamental explanations; authors even explain in details ==-== how to reproduce the paper; documentation is not provided; ==-== documentation is lacking the following basic information;... ==+== K. If the artefact reached the "functional" level, meaning that it ==+== seems to function correctly, can the artefact be easily reused ==+== and/or repurposed? ==-== Rate from 0% to 100%, where 0% means IMPOSSIBLE TO REUSE; 20% VERY ==-== LIMITED REUSABILITY up to 100% meaning ARTEFACT VERY EASY TO ==-== REUSE/REPURPOSE. ==-== Choices: ==-== 1. 0% ==-== 2. 20% ==-== 3. 40% ==-== 4. 60% ==-== 5. 80% ==-== 6. 100% ==-== Enter the number of your choice: (Your choice here) ==+== L. Comment about the degree of reusability/repurposing ==-== E.g., I already have some ideas how to reuse it; the artefact does ==-== not sufficiently feature-rich to be reused; I can see ==-== opportunities to use it for; the artefact is functional but ==-== reusing it is kind of a hassle... ==+== M. If the artefact reached the "reusable" level, meaning that it ==+== seems to both function correctly, and can be reused, was it possible to ==+== replicate the author's results? ==-== Rate from 0% to 100%, where 0% means IMPOSSIBLE TO REPLICATE; 20% VERY ==-== LIMITED REPRODUCIBILITY or REPEATABILITY up to 100% meaning VERY EASY TO ==-== REPLICATE RESULTS. ==-== Choices: ==-== 1. 0% ==-== 2. 20% ==-== 3. 40% ==-== 4. 60% ==-== 5. 80% ==-== 6. 100% ==-== Enter the number of your choice: (Your choice here, this question is optional) ==+== N. Comment about the degree of reproducibility/repeatability ==-== E.g., I managed to replicate some/all of the author's results, ==-== I was able to replicate the results, but slight changes led to significantly ==-== different results,... (This question is optional) ==+== O. Artefact Weight in the Research Paper ==-== Considering the portion of the paper supported by artefacts ==-== (included other artefacts not proposed by the authors), what share ==-== the key results therein are clearly directly linked to the ==-== provided artefact? Please choose between 0% and 100%. Where 0% ==-== means no key results are directly related to the artefact and 100% ==-== means that all of the key results therein are clearly directly ==-== obtained with the artefact. ==-== Choices: ==-== 1. 0% ==-== 2. 20% ==-== 3. 40% ==-== 4. 60% ==-== 5. 80% ==-== 6. 100% ==-== Enter the number of your choice: (Your choice here) ==+== P. Badge Recommendation based on the following ACM Badge definitions ==-== **Artefacts Available:** Author-created artefacts relevant to this ==-== paper is publicly available. A DOI or link to this repository ==-== along with a unique identifier for the object is provided. ==-== **Artefacts Evaluated – Functional:** The artefacts associated ==-== with the research are found to be documented, consistent, ==-== complete, exercisable, and include appropriate evidence of ==-== verification and validation. ==-== **Artefacts Evaluated – Reusable:** The artefacts associated with ==-== the paper are of a quality that significantly exceeds minimal ==-== functionality. That is, they have all the qualities of the ==-== Artefacts Evaluated – Functional level, but, in addition, they are ==-== very carefully documented and well-structured to the extent that ==-== reuse and repurposing is facilitated. In particular, norms and ==-== standards of the research community for artefacts of this type are ==-== strictly adhered to. ==-== ==-== See Artifact Review and Badging for details. ==-== Choices: ==-== 1. No Badge ==-== 2. Artefact Evaluated - Available ==-== 3. Artefact Evaluated - Functional ==-== 4. Artefact Evaluated - Reusable ==-== 5. Artefact Evaluated - Replicated ==-== Enter the number of your choice: (Your choice here) ==+== P. Comments ==+== Findings and comments that were not covered by previous questions. ==+== End Review