Important note For deadlines before April 27, we solicited submissions in the IEEE format. For all later deadlines, please use the ACM format as described below.
All submissions must come in PDF format and conform, at time of submission,
to the ACM Proceedings Template (LaTEX users must use
\documentclass[sigconf,review,anonymous]{acmart}).
Technical Research papers and Experience papers must not exceed 10 pages plus up to 2 pages that contain ONLY references. New Ideas papers must not exceed 6 pages (including figures, appendices, AND references). Please ensure your paper incorporates the following items. This is not a complete list and does not replace the ACM Formatting Guidelines, but it serves as a list of common issues:
- Prepare your paper as a letter-size PDF document.
- Embed all fonts.
- Use only scalable font types (e.g., Type 1, TrueType). Bitmapped font types (e.g., Type 3) are not acceptable.
- Use either vector graphics or at least 300-dpi graphics.
- Caption every figure and table and reference them in the text.
- Do not use footnotes or references in the abstract.
- Avoid color in the regular body text. Color is allowed in figures, tables, listings, and in non-body text, such as source code, XML snippets, etc.
- Make sure the first reference does not overlap the heading "References".
- Balance columns on the last page.
Make sure the paper is readable on a black/white printout. Reviewers are not required to try to decipher tiny fonts in figures or nuances between colored lines that print as the same shade of gray.
Important: submissions will be checked for compliance with the required formatting criteria. Papers not conform to the required style (e.g., using tighter margin, exceeding the length) may not be considered and may be therefore desk rejected.
ASE 2018 will employ a lightweight double-blind review process. The papers submitted must not reveal the authors' identities. However, the author identities will be revealed when the reviewers submit their reviews and start discussing the pros and cons of the submissions. While there can be pros and cons of revealing identities during the discussion phase (i.e. one option can be to have a totally blind process), we decided to adopt such a lightweight process (as ASE 2016 did) to avoid that a paper is accidentally reviewed by a reviewer having an undeclared conflict of interest.
The authors must make every effort to honor the double-blind review process. In case of questions, please contact the PC Chairs.
Please check off the following items:
- Omit authors' names and institutions from your title page.
- When you cite your own work, refer to it in the third person. For example, if your name is Smith and you have worked on automated bug repair, instead of saying "We extend our work on information retrieval for finding and removing earwigs [0]," you should say "We extend Smith's [0] work on information retrieval for finding and removing earwigs."
- There may be cases in which the current submission is clear follow up of one of your previous work, and despite what is recommended in the previous point, reviewers will clearly associate authorship of such a previous work to the current submission. In this case, you may decide to anonymize the reference itself at submission time. For example: "based on previous results [10]" .. where the reference is reported as "[10] Anonymous Authors. Omitted per double blind reviewing." In doing so, however, please make sure that the ASE 2018 submission is self-contained and its content can be reviewed and understood without accessing the previous paper.
- Do not include acknowledgements of people, grants, organizations, etc. that would give away your identity. You may, of course, add these acknowledgements in the camera-ready version.
- In general, aim to reduce the risk of accidental unblinding. For example, if you use an identifiable naming convention for your work, such as a project name, use a different name for your submission, which you may indicate has been changed for the purposes of double-blind reviewing. This includes names that may unblind individual authors and their institutions. For example, if your project is called GoogleDeveloperHelper, which makes it clear the work was done at Google, for the submission version, use the name DeveloperHelper or BigCompanyDeveloperHelper instead.
- Avoid revealing the institution affiliations of authors or at which the work was performed. For example, if the evaluation includes a user study conducted with undergraduates from the CS 101 class that you teach, you might say "The study participants consist of 200 students in an introductory CS course." You can of course add the institutional information in the camera-ready. Similar suggestions apply for work conducted in specific organizations (e.g., industrial studies). In such cases, avoid to mention the organization's name. Instead, you may just refer the organization as "Org" or "Company", etc. When appropriate and when this does not help too much in revealing the company's name, you might mention the context (e.g., financial organization, videogame development company, etc.).
- If you think that supplementary material, such as links to replication packages, additional information, or released tools, is important for reviewers, do not omit it from the submission for promise them only for the camera ready version. Reviewers are never required to assess supplementary material, but they often do and won't have a chance if a release is only promised but not provided. There are different strategies to provide supplementary material: (1) ASE 2018 allows an optional appendix to include additional anonymized material. (2) Some material can be made available on anonymized external packages, for example, under a fresh Github account or dropbox link. (3) If anonymization is not reasonably possible, include a scrambled link (e.g. bit.ly) and add a warning "following the link might reveal the author's identity".
- Avoid linking directly to code repositories or tool deployments which can reveal your identity. You may post anonymized links (with a warning that following said link may reveal authors' identities), include links to anonymized code or deployments. When creating such repositories, a good practice can be asking somebody in your team to test the anonymization of the repository and of its content. In case anonymization is difficult to be achieved and you still want to provide availability of data/tools, you can simply state that you will link to the code or deployment in the camera-ready version.
See the double-blind reviewing FAQs below for more information.
Q1: Why is ASE 2018 using double-blind reviewing?
Studies have shown that a reviewer's attitude toward a submission may be affected, even subconsciously, by author identity. We want reviewers to be able to approach each submission without such involuntary reactions as "Barnaby; he writes good papers" or "Who are these people? I have never heard of them." For this reason, we ask that authors omit their names from their submissions, and avoid revealing their identities through citations and text. Many systems, security, and programming language conferences use double-blind reviewing and have done so for years (e.g., SIGCOMM, OSDI, IEEE Security and Privacy, SIGMOD, PLDI). Software engineering conferences are gradually starting to adopt this model. This year most of the software engineering conferences (ESEC-FSE 2017, ISSTA 2017, ICSME 2017 MSR 2017, ICPC 2017), as well as upcoming editions of ICSE (2018) adopt double-blind reviewing too. This process will be cooperative, not adversarial. While the authors should take precautions not to reveal their identities (see answer to Q4 below), if a reviewer discovers the authors' identities through a subtle oversight by the authors, the authors will not be penalized. A report about the experiences at ASE 2016 is available: https://arxiv.org/abs/1709.01609
Q2: Do you really think blinding works? I suspect reviewers can often guess who the authors are.
Reviewers can sometimes guess the authorship correctly or work it out through references or supplementary material, though studies show this happens less often than people think, including reviewers at previous ASE conferences (https://arxiv.org/abs/1709.01609). Still, imperfect blinding is better than no blinding at all, as it avoids many forms of priming and implicit biases. Even if all reviewers guess all authors' identities correctly, double-blind reviewing simply becomes traditional single-blind reviewing.
Q3: Couldn't blind submission create an injustice if a paper is inappropriately rejected because a reviewer is aware of prior unpublished work that actually is performed by the same authors?
The double-blind review process that we will be using for ASE 2018 is lightweight in the sense that author names are revealed once reviewers have submitted their reviews and start discussing the pros and cons of the submission. In this phase, the authors' previous work can be explicitly considered.
Q4: What exactly do I have to do to anonymize my paper?
Your job is not to make your identity undiscoverable, but to make it possible for our reviewers to evaluate your submission without knowing who you are.
If you have a concern that particular information is particularly easy to trace to you, consider adding a warning to reviewers in a footnote, such as "Note for reviewers: searching the commit logs of the github projects we used in our evaluation may reveal authors' identities."
Q5: I would like to provide supplementary material for consideration, e.g., the code of my implementation or proofs of theorems.
How do I do this?
In general, supplementary material should also be anonymized.
Please make your best to avoid (i) having your names/affiliations in artifacts' metadata (e.g. PDFs, spreadsheets, other documents); (ii) having contributors' names in source code.
To create a repository, you could use an anonymized cloud account (i.e., created with a username not clearly attributable to the authors), or similar solutions.
If the code or the repository cannot be anonymized easily, provide an anonymized url (such as using a URL shortener like http://bit.ly) with a prominent warning to reviewers that following the link may unblind them. Simply stating that a link will be added as part of the camera ready version has little value to a reviewer.
Q6: I am building on my own past work on the WizWoz system.
Do I need to rename this system in my paper for purposes of anonymity, so as to remove the implied connection between my authorship of past work on this system and my present submission?
Not necessarily.
If knowing the name of the system reveals your identity or institution, then you should use an alias system name in the submission.
For example, a paper on a modification to a proprietary system (e.g., Visual C++, or a closed-source research project) implicitly reveals the identity of the authors or their institution, and that paper should use an alias name for the system.
In your submission, point out that the system name has been anonymized.
If you have any doubts, please contact the PC Chairs.
By contrast, if the system is widely available and open-source, there is no need to rename the system.
Q7: Am I allowed to post my (non-blinded) paper on my web page?
Can I advertise the unblinded version of my paper on mailing lists or send it to colleagues?
May I give a talk about my work while it is under review?
As far as the authors' publicity actions are concerned, a paper under double-blind review is largely the same as a paper under regular (single-blind) review.
Double-blind reviewing should not hinder the usual communication of results.
But, during the review period, please don't broadcast the work on social media.
Also, to the extent to which this is possible, please avoid to publish the preprint of your work (e.g., on arXiv or on your website) until it is accepted for publication.
Q8: Will the fact that ASE is double-blind have an impact on handling conflicts of interest?
Using double-blind reviewing does not change the principle that reviewers should not review papers with which they have a conflict of interest, even if they do not immediately know who the authors are.
Conflicts of interest are identified based on the authors' and reviewers' names and affiliations, and they can be declared by both the authors and reviewers.
Q9: What should I do if I learn the authors' identities?
What should I do if a prospective ASE author contacts me and asks to visit my institution?
If at any point you feel that the authors' actions are largely aimed at ensuring that potential reviewers know their identities, you should contact the PC Chairs.
If you are unsure, contact the PC Chairs.
Otherwise you should not treat double-blind reviewing differently from regular single-blind reviewing.
You should refrain from seeking out information on the authors' identities, but discovering it accidentally will not automatically remove you from reviewing a paper you have been assigned.
Use your best judgment and feel free to contact us with any concerns.
Q10: How do we handle potential conflicts of interest since I cannot see the authors' names?
We will handle conflicts of interest as usual through the conference management system, which will ask you to identify conflicts of interest before bidding. For this purpose, a list of all authors and affiliations will be presented. Please see the related question (Q8) applied to authors to decide how to identify conflicts.
This FAQ is based on several iterations of PLDI and SIGMOD guidelines for double-blind reviewing.
