Ok, I meant to do this post about 3 months ago, a writeup 2 months ago and ended up just posting the video below over a month ago. I simply have not had time, and for that I apologize. I made the video because I could simply talk out what was in my head. I used a presentation I was still in the middle of writing. It was also 3 am after a long day. Not the best recipe for success.
I will try to get an additional post on other aspects of the PyCon 2009 program selection, but for now we will focus on the submitted proposals, and the process used to select the best talks from those proposals.
Identify the Champion
We are using a process based around the ‘Identify the Champion‘ paper by Oscar Nierstrasz. I highly recommend reading this document in detail. It does not dictate how a program committee will do its work, but rather sets a framework from which many different systems can be developed to meet the specific needs of the conference or group in question. The central theme is breaking down the problem by ensuring the proper people (the ‘champions’) drive the process. I do want to hilight one fact before we get into the process:
Reviewer Votes do NOT determine if a Proposal is accepted or declined
The program committee as a whole decides which proposals will be accepted. We will get back to this later, but I want to dispel this idea of ’scores’ on proposals up front. With that said, lets start…
Starting at the End
Humor me for a moment as I work backwards from the end result to the beginning. The last thing we do is announce the program schedule to the world. Before we do that we need to get some feedback from the proposal authors, and before that we need to let them know what decision the program committee made. The last 4 things the Program Committee does (in reverse time) are:
- Announce the schedule to the world
- Process feedback from the authors
- Send accept/decline decisions to the authors
- The Program Committee has reached consensus on the accepted talks
The way the PC reaches consensus is by having a meeting or set of meetings to determine which talks will be accepted. The entire PC process is about making those end meetings manageable, and providing the best information for making the best decisions.
Program Committee Consensus Meeting(s)
The big piece of work here is the end PC meeting. Last year we received 140+ proposals, but only had space/time for 62 talks (and this was after reducing the number of 45min talks to just 4). It is not about selecting the best talks, but about providing the best overall program. To do that we need to break the problem down, and work as a team. Bullet point time:
- All proposals will be discussed and reviewed at the meeting
- The meeting needs to be organized (or it will devolve into chaos)
- No one person can review all proposals in detail
- We need ways to break down the problem and categorize the talks by some criteria
- Its all about the Champions
All about the Champions
- Champions are people who have agreed to argue for or against a particular proposal
- Champions get to speak first when discussing a proposal in the end meeting(s)
- Champions get the rest of the PC up to speed on a particular proposal
- Champions need to have an active interest in the proposal and topic of the proposal
- A Champion need not be a domain expert, but it is nice to have at least one domain champion per proposal
- All proposals should have at least one champion for or against (but this is not a requirement).
- Champion status is used to break down the proposals into interesting categories (below).
Identify the Champions
We want the proposals to have good review coverage. For this reason we want at least 3 people to review each proposal. This helps prevent gaming of the system, and we want different viewpoints. While a domain expert can provide crucial insight into a proposal, that expertise can actually be a liability for beginner talks in particular. Someone who is interested in a topic, but has limited exposure to it may be a better champion for a beginner introductory proposal than someone who is an expert in the field. You would like to get both types of reviewers. As described in the document, we let the reviewers decide which talks they will champion (for or against).
Proposals are not just reviewed by people who are willing to champion a talk. The job of reviewers is to review the proposals and to work with the authors, nut just to be champions. A reviewer can provide crucial information on a proposal, but not feel strongly enough one way or another to champion said proposal.
Writing Reviews
A review consists of information on the proposal for the general program committee, and if you will champion the talk. We use the familiar ‘+1/+0/-0/-1′ voting system, but those values might not mean what you think they mean:
- +1 I will champion for this proposal
- +0 I like this proposal, but will not champion it
- -0 I dislike this proposal, but will not argue against it
- -1 I will argue against accepting this proposal
As such the vote value is about championing the talk, and is not a score. Scores do not work (see the document for supporting materials). While this vote is valuable information, the most important information is provided in the text of the review, and communication with the author(s). In order for the PC to make the best decisions we need more information than just what is present in the proposal. Author bios, previous history, command of the topic, and other information is crucial. It is the job of the proposal reviewers to gather this additional information for the greater PC. It is the job of the Champions to help communication this information in the end meeting(s) answering questions and clarifying the reviews when needed. The PC should also communicate with the proposal author(s) for clarification and answering questions when needed. This should happen before the end consensus meetings, as part of the review process.
Breaking Down (the proposals)
We use the vote values to identify interesting groups. Many talks will occur in multiple groups. The program committee will iterate over the proposals in a number of passes, first identifying work left to be done, then getting familiar with the proposals, and finally trying coming to consensus on which few to accept. These groups are usually iterated over before and during the meeting(s) as follows:
- Missing Reviews (those with <3 reviews)
There should not be any proposals in this category for the end meeting(s). If there are, it is an indication of other problems in the Program Committee.
- Needs Champion
These are talks which do not have a champion for or against. This does happen, and is fine. If there are many talks in this category, it might be an indication of some other problems in the Program Committee or the pool of proposals. These should be reviewed first to see if any reviewers are on the fence about being a champion.
- All Negative
Just because all the reviews are negative, does not mean that it is a bad proposal. These need to be scrutinized for hidden gems, and undue bias.
- Mixed
This is the most interesting grouping, especially those talks with champions both for and against. This can be an indication of a ‘hot’ or controversial topic. Some of the best talks fall into this category.
- All Positive
The proposals which no one objects to are left for last.
From there it is down to picking the best proposals which give the best coverage on interesting topics. The review text, and communication with the authors becomes crucial for making decisions at this stage.
Communicating with the Authors
Because reviews take time, and will often require authors to answer questions and make clarifications, authors will be allowed to edit their proposals well after the proposal submission deadline. Reviewers and Authors need to have a recorded means of communicating and optimally this will be recorded as part of the proposal (but that gets into the software part of things, not the process, and thats a different post). The proposals will need to be locked down at some time before the end meetings so we are not accepting moving targets.
Back to the Beginning
- Proposals reviews have their final edits
- Proposals have their final edits
- Proposal Deadline (no more new proposals)
- Call for proposals goes out
- Program Committee is formed
Further Reading
The Horrid Video
