Just wanted to give a status update to all those who are wondering what is going on with the PyCon US 2009 program committee.
We have recieved 109 proposals. (NOTE: there is now a dynamic image URL so you can embed the above graph in your blog! You need to right click as it uses redirect.)
This number is lower than last year, but that is to be expected. Why? Because:
- We did not advertise like last year (where we actually paid for professional help to get the call for proposals announced all over the place).
- We did not announce it in popular IRC channels like in past years
- We started collecting proposals later than last year
- We had a much shorter submission period this year (even before last years extension)
- We did not extend the submission period like we did last year
- We did not put out regular announcements of the deadline approaching like we did last year
- There have been a TON of new regional and international python conferences this year
- The economy?
Seriously, I am shocked that given all that, we still got the number of proposals we did. I am also ecstatic to say that we have 39 reviewers from a broad cross section of the python community. We also have a number of invited speakers. These speakers were selected from the feedback forms from previous years and from general feedback from the greater community. The organizers and the Program Committee reviewed feedback from 08 (the surveys, blog posts, reddit, and mailing lists, IRC logs, etc) in an attempt to discover what things worked last year, the things which did not, and how to do better.
So what is happening now?
The program committee is activly commenting on and reviewing the proposals. At the time of my writting this, there have been 170 comments. We use comments to communicate with the authors for clarifications and to suggest changes. Authors can still edit their proposals and we are working towards reaching a critical mass. As of this post there are only 132 reviews written. That number will grow to well over 400 before we are done. That is a monumantal effort for the program committee, and it is just the tip of the iceburg.
What happens after that?
Once the comment period is over, the PC will will head into the home stretch for reviewing. The proposals will be made read only and the PC will make final review edits, and ensure that there are no proposals which are under reviewed. That is where the fun begins; if you consider trying to get 40 people to all agree on anything fun. I won’t bother rehashing how that happens without thing degrading into chaos except to say that should not given the processes used.
What do you think so far?
There has been quite a bit of discussion on the pycon-pc list on how reviews should be written, how best to communicate issues with the authors via comments, and how to manage the PC effort. Even though the core of the process is stable, every year it is different. One complaint last year was that there was not enough transparency with this process. I have tried to address some of this in blog posts, etc. and this is part of said effort. One missing element is early public discussion on the PC from submitters, reviewers, and the general community. So lets start some of that right now.
If you are so inclined please leave a comment on this post on your thoughts about the process thus far. Please mention if you are a reviewer or proposal author for this year, for a previous year, a past PyCon US attendee, or just someone in the general community.
There are some very strong opinions out there, so please do not get upset just because you don’t agree with someone elses opinion and please keep it civil.
UPDATE: Yes the registration system had issues, we are talking about the program committee. No, authors were not hidden from the reviewers last year, that is a myth I have attempted to debunk time and again. Yes, we all know the sponsor lightning talks were a mistake, that horse is pulp; I will delete any comments brining that up. Same goes for any comments with profanity. *sigh*
First of all, congratulations for all your work, and that of everyone involved in organizing Pycon. While I have not participated in Pycon’s organization, I have a fairly good idea of all the work required to make such a conference work and I am really impressed with how smoothly everything is run.
I am curious about one point which you did not address: How do you handle conflicts of interest? If Bob, who volunteered as a reviewer, submits a talk, how do you ensure that he is fair in his assessment’s of a talk by Jane on the same topic? Perhaps Bob’s proposal was written after he read Jane’s, etc. Just curious…
@André
I have never witnessed any improper conflict of interest, and I don’t think there are any in the community which would try to subvert PyCon… but…
Conflict of interest is a difficult problem, and there are different aspects of it to deal with.
Foremost we need to trust our reviewers. What I mean by this is that there is no way to be 100% sure there are no conflicts of interest at play. There are no automated or software solutions to solve this problem. It is inherently a human based issue. As such draconian procedures will only cause problems. Most people are more sensitive to their own perceived conflicts of interest; that is we judge ourselves more harshly than others would on average. We need to trust that our reviewers will ‘do the right thing’ as it is impossible to prevent it otherwise.
With that said there are checks and balances.
There are two primary forms of acting on conflict of interest:
A. constructive (author is a known associate, etc)
This is the most common, as the python community is not that large, especially with domain topics like web2py or jython. This is not a bad thing as we want those domain experts reviewing these talks. Also information like ‘this author is a friend of mine. he gave the talk at the local users group and it was well received.’ is information crucial to the program committee. Here we ask that reviewers detail their conflicts of interest in the review.
B. Subversive (collusion and sabotage)
This includes the scenario you bring up. These are situations where a person receives some sort of benefit from seeing a talk accepted or declined. Here there are three checks to try to detect when this happens.
1. At least 3 reviewers
Every proposal must have at least 3 reviews. It is much harder to improperly review a proposal when at least two others are also reviewing the same talk. Many of these reviewers are prominent in the python community. Passive pier pressure can be a powerful motivator.
2. Reviews do not determine acceptance
We do not use a computed score when writing reviews. The mark given is just to help break down the talks into interesting groups. As such it is impossible for someone to kill off a proposal. The PC as a whole discusses the proposals by breaking them down into successive groups. Part of that is breaking them down by overlapping and intersecting topics. It should be obvious if a champion speaks up against a talk then turns out to be the author of of a talk in that same group.
3. Authors get to see the reviews and provide feedback
Before the accepted talks are announced to the general public, the proposal authors get to see the written reviews and provide feedback. Many times they are in the best position to identify when there is a subversive conflict of interest.
At the end of the day you need to trust the community.