Response to Reviews

3 times “reject” and one “strong reject”. I braced myself for a negative result of submitting my paper “Purely Functional Structured Programming” to the ESOP 2011 conference, but this is way below any review score I have previously received for any of my papers. So what went wrong?

First, I clearly misunderstood the scope of the conference. On its conference site it says:

ESOP 2011 is the twentieth edition in this series and seeks contributions on all aspects of programming language research including, but not limited to, the following areas:

Programming paradigms and styles: functional programming, object-oriented programming, aspect-oriented programming, logic programming, constraint programming, extensible programming languages, domain-specific languages, synchronous and real-time programming languages.

They forgot to mention:

  • If you present an operational semantics, don’t make it too operational. DO NOT PROVIDE ANY WORKING CODE. If you do, it is great, but I, the reviewer, cannot be bothered to actually run the code to confirm that an example I am wondering about is correct. Even that last one on page 11 (EDIT: for n == 0, the result in the last example degenerates from a list of length 1 to an integer, so I guess I owe reviewer 1 an apology, if he/she was referring to this) .
  • If you present a new paradigm of coding, focus on the technical difficulties of this paradigm. If there are no major technical difficulties because the paradigm just works, invent new ones. Oh, it is blindingly obvious that Mini Babel-17 code can be easily transformed to pure Standard ML code, especially after the whole “shadowing” stuff? Not to me. Thats why I designate my confidence level as “Expert”.
  • If you have something important to say, say it in 20 pages, even if you could say it  in 12. You can fill up the remaining space easily with inference rules that have cryptic symbols above and below them.
  • Present a type system. If you cannot say what you want to say by dressing it up as a type system, better refrain from saying it in the first place.

Well, obviously, I have written the above lines in a frustrated state of mind. I should say that I am thankful to the reviewers for investing their time in reviewing my paper. And I am. Clearly the paper has failed to communicate what it is all about. It is not about the technical difficulties on how to hide mutable state inside functions that are pure when watched from the outside. It is not about presenting an alternative way to Haskell monads when dealing with mutable state. It is about how I can easily and naturally use “For” and “While” loops in a purely functional setting in those cases where I feel that these loops are the best way to express my intent. It is about how


let
val a = ....
in
let
val b = ...
in
...
end
end

is clearly less readable than


val a = ...
val b = ...
...

It is about how the inventors of Scala clearly understood this, but failed to recognize that then


val a = ...
val a = ...
...

should be allowed too (via the correspondence to let-constructs). And it is about how this seems to be a minor issue, but that this minor issue prohibits the language design to get the integration of loops with purely functional programming right.

Anyway. Words are cheap. Back to coding again. A full implementation of Babel-17 v0.21 is close.

Advertisements

10 Responses to “Response to Reviews”

  1. Jane Doe Says:

    I can understand your frustration, but frankly speaking the paper is nowhere close to a state where it could possibly be accepted for a competitive conference like ESOP (I had a look at the arXiv version). I am not talking about the contribution now, but the exposition does not meet scientific standards (in my opinion) and lacks important details, which is a shame considering that you had 8 pages left.

    • phlegmaticprogrammer Says:

      Scientific standards and important details: What are you missing?

      The reviews make clear that they just don’t get it. I don’t think it helps drawing more trees if people cannot even see the forest.

  2. Jane Doe Says:

    It might be true that the reviewers just “did not get it”, but the key question is “why”? Some statements in your paper are very controversial, and they have no backing (for example, through references or a proper discussion). You do not even motivate what it is all about, and why you consider your contribution important, and why other people should consider the problem important as well, just something vague like “Let’s make PFP more mainstream by integrating SP … This is what this paper is about”. To stay with your image, it is your duty to write a scientific text that forces the reader to see the forest.

    A few points that directly come to my mind (again, not judging or even discussing the scientific contribution):

    1) Let us start with the introduction, which is too colloquial for scientific writing, it lacks proper arguments (“because it smells of side-effects” is no argument), it over-uses exclamation marks, etc. This introduction is a perfect start to have your paper rejected, some people might even find it offending.

    2) The transition from Section 1 to Section 2 is abrupt, there is no supporting text to guide the reader through what he has to expect and what the whole paper is about.

    3) The formatting is terrible, it looks scruffy and not “pleasing” at all. Nice if you want your paper rejected.

    4) The listings on pages 8-10 are pretty much useless as such, would be ok for an appendix, but the section lacks motivation, details and support text.

    5) The related work is too short, and there is no proper comparison that goes beyond folklore knowledge. The literature references contain mostly standard textbooks, but that’s it.

    Personal hint: If you submit to ESOP, CAV, POPL, etc. then send the paper to colleagues and friends for early extremely critical feedback, no matter if they have a background in programming languages or not. If both experts and non-experts are pleased with your work, you might have a chance. If the non-experts didn’t get it, that already indicates a problem.

    There are many excellent papers in major conferences, just have a look at the recent POPL, PLDI and ESOP contributions and try to compare their writing to your introduction, and your related work. The related work section alone forces any serious reviewer to reject your paper.

  3. Jane Doe Says:

    To add some backing to my comment, let’s have a look at the first review, which states: “It is interesting as part of a complete language design. But I don’t think there is enough here for an ESOP paper. My summary above is quite short but I think it captures the whole of the paper; I don’t think it would be possible to summarise a typical ESOP paper so briefly.”

    This is straight to the point. To get into a major programming languages or verification conference (ETAPS-level), you need to have either a single killer contribution (which you do not have) or 3 to 4 serious and clearly stated contributions (which you do not have either). As I said, your paper is nowhere close to a state where it could possibly make it into ESOP.

    I’ve served as a referee for ESOP, TACAS, etc. in the past and there are so many reasons to reject a paper. Poor presentation, lack of experimental evaluation, not enough (technical) meat, poor comparison to related work, no future perspectives (research that might emerge from your contribution) etc. Failing in a single category typically leads to a reject.

    • phlegmaticprogrammer Says:

      You are right, contributions to major conferences are judged by the criteria you mention. What is also interesting about your answer is that you spend a lot of lines explaining why the form (“not judging or even discussing the major contribution”) of my paper is appropriate. When I grew up, I thought that science is about content, not form. Now, almost 3 years after having obtained my PhD, I still think that, but I realized that mostly, science as displayed at conferences is for careers, and not advancement of knowledge. 90% of the papers accepted at ESOP’11 will be forgotten the day after the conference ended and the conference proceedings have been published. I hope that my contribution will be relevant long after that.

      You say the program listing is useless. It is a program written in Standard ML, which has a formal semantics. Therefore the program listing also has a formal semantics. You can even execute the program, making it an “operational semantics”. I have even provided an interpreter separate from the ML program that accepts a much nicer syntax. I don’t think that it is possible to be any more precise than I have been. I even went through the trouble to carve out a subset of Babel-17, Mini Babel-17, to focus on the topic (linear scope) I wanted to come across. With hindsight, this might have been a mistake, because arguably people fail to see the potential of the approach when they see only a piece of it. Most scientists don’t have vision. Only the great ones do. (at least I hope the great ones do … looking at history you might become convinced otherwise)

    • phlegmaticprogrammer Says:

      Nevertheless, I understand that form is important if you want to convince people. I have held scientists to a higher standard than ordinary folks, but this is of course a mistake.

      My main goal will be to bring Babel-17 and its documentation into a form that convinces programmers. The coding most scientists do is negligible anyways, so they are not my target audience and I am not going to spend more energy trying to convince them. If Babel-17 has success with programmers, conference papers about it will follow easily. If Babel-17 doesn’t have success with programmers, then conference papers don’t matter.

  4. Jane Doe Says:

    There is a famous quote by Amir Pnueli, whose scientific work and style of presentation I wholeheartedly admire: Theoretical computer science is all about “notation, notation, notation”. I believe that requiring a high standard in terms of presentation is part of the ethics of science: it makes things accessible, it clarifies and does not obfuscate, it influences the impact. In this sense proper presentation belongs to science as much as the technical contributions do.

    “I don’t think that it is possible to be any more precise than I have been” … but it lacks support text, motivation, intuition, etc. That’s why it’s useless in the current form: it may be correct and it may be important, but in the current form it is useless.

  5. phlegmaticprogrammer Says:

    I think we have a different understanding of what “useless” means. That’s probably why I left academia, and why you are still in it.

  6. Well.. Says:

    At least you get constructive feedback in your reviews. In systems-y conferences, a one or two sentence “i don’t like it. reject.” is fairly common.

    Its up to you whether to ignore it out of hubris or improve your presentation.

  7. phlegmaticprogrammer Says:

    I do not mind improving my presentation. But honestly, a little hubris is necessary, because if all we had today in science was stuff that made it through its first peer review, we would be nowhere.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: