Dougma (dŭg·mə) n.

After finally being done with the talk proposals on Sunday, I decided to take a break and read some of those python blog thingies to see what the clean people are up to. I was hoping to find a few topics of minutia I could loose myself in. What I found were a ton of posts on the new Google Chart API. Now I have been looking for a good chart solution for quite some time. I even discussed certain options over at Marty Alchin’s blog. I have looked at DojoX Charts, and Open Flash Chart, and many others. Dojo has the best charts, but the API is a PITA to figure out. the Doc is mostly auto generated from the code, and the samples are next to useless. I know I can do great things with it, but I recently spent 3 hours on it and got nowhere. Open Flash Chart got me up and running in no time, and has some nice python bindings, but its a flash based solution. So when I saw the Google API, I just dove into it’s doc to see what is up, and forgot all about the blogs. Lets see what happened next shall we?…

Read the rest of this entry »

This years PyCon Program Committee really has really had a very difficult task this year. Not many people know what the Program Committee goes through when putting together the final list of talks; mainly because in the past there has not been much to it. That was not the case this year, and I want to shine some light on what happened, because I feel the community should know about the efforts of a few for the benefit of the many. Now this is not an official record of events, and it is shaded by my own bias and prejudices. I am sure others will have different views, and some different recollections. I will try to keep things to the facts, and that means graphs!!!

First things first, I should explain the basis of the proposal system and review process. Things are broken down into phases, and we kinda adhere to them, sorta. We accept proposals, we help authors finalize proposals, we vote on proposals, then we decide which ones are accepted as talks; usually in that order. to be honest this was my first year being really active in the entire process. Last year I reviewed a number of proposals, but was only called upon to defend one during the final review meeting (more on that later). I was more concerned with the software behind the proposal system, and fielding feature requests from the authors and reviewers alike. this yea, with the shear number of proposal, I ended up doing much more than I had planned; that is true of everyone on the committee.

We form a program committee by gathering up some suc… err.. volunteers and call them reviewers. Normally there is not too much work as a reviewer. You write about 15 reviews, give some scores, and maybe defend some talks in a meeting on irc. As proposals are submitted, 3 reviewers are randomly assigned. Reviewers have the option of opting out of talks they don’t feel comfortable reviewing for whatever reason, and someone else is assigned. Reviewers can review any talk they want, not just those they were assigned to. The important part is that each proposal gets at least 3 reviews. Well this year we had 140+ proposals, 18 reviewers. Many reviewers made more than 40 reviews, almost all made over 30.

Reviews get a score; +1, +0, -0, -1. These are not numeric values to be summed. Talks do not automatically get approved because they get all positive review scores. I wont go into a full breakdown, but +1 means that you are willing to champion a proposal. You are willing to speak up for it as the end review meeting and argue for its inclusion. +0 means that you like the proposal but are not willing to argue for it. The negatives are for arguing against. We like to have all proposals to have a champion. After the submission deadline, reviewers communicate with the proposal authors and try to get the proposals into a state where they are worthy of championing. The idea is to help the community put on the best conference possible. this means helping the authors, not just rate them. Scores are given on individual merits. If its a good proposal and deserves being championed, it does not matter that there is some other proposal that is somehow ‘better’. they should both be championed and discussed at the end review meeting. Proposals with -0 and even -1 scores might get chosen over those with two champions. The idea is to put together the best overall list of talks. The reviews and scores are just the framework used for the real decision making process. They are helpers, not the end decider’s.

With that lets look at some numbers! We had 140+ proposals, 18 reviewers, almost 450 reviews, and almost 200 comments (communications with authors). That is correct, almost 450 reviews written by 18 people. Also over 200 proposal edits after submission, and almost 50 review changes due to those edits. The only statistics from last year I have are 101 proposals, 22 reviewers, 340 reviews, and 30 comments. So lets take a look at some graphs!!

Thats right, almost 1300 change history entries! That is a mountain of work, and all that before we even get to the real job of deciding which talks to accept! I shoul dbe doing other things, but I could not resist a chance to use the new google API That very last uptake in proposal edits is mainly the program committee deciding which talks are accepted. The uptake just before that was in response to a flurry of comments from reviewers trying to get everything done before the proposal edit deadline. Things flattened out over the week however, so the late deadline was largely unnecessary for author edits, but was crucial for the reviewers to respond to those edits and update reviews, or in many cases, make them, now that more information was available. With that work done, we could focus on championing our respective pet proposals and making decisions.

Now in past years the program committee would tackle those talks with mixed reviews and start arguing about like talks with overlap. In 2006 there were about 20 talks which had champions trying to convince others that they should go in over other talks, and the remainder everyone pretty much agreed upon quickly. Every talk was discussed, but there was not much convincing that needed to be done. This year things were different. We had so many good talks, and just 64 open spaces to fill. In 2006 we had a single 3 hour chat, which also included much of the scheduling work. This year we had 4 IRC meetings, totaling over 10 hours, and multiple e-mail threads. We covered over 100 proposals in a fare amount of detail, and sweated over the final 15 accepted for over 3 hours. (Not sure how many in total were compared and contrasted, and argued in that time; must have been over 40). There were no real arguments, but it was not an easy run. I do not thing anyone is 100% satisfied with the end selections, which I believe means we struck a good balance. I will be blunt and say that we had to decline proposals that in any previous year we would have begged for.

I think the main reason things ended up going so smoothly was because the Program Committee head, Ivan Krstić, was keeping the rest of us on track, and had things quite well organized during the meetings. The first IRC meeting on the 2nd, Ivan almost didn’t make it due to internet connectivity in Uruguay! That’s right, he was managing the first OLPC roll out, and connected up to IRC to run the meeting. I seriously have no clue when he sleeps. He was down right tenacious in making sure we moved on and did not spin off on a tangent for too long. this meant deferring some decisions until later, which worked quite well. Ten hours of meetings is running smoothly? Yes. Given the talks, ten hours is nothing. That breaks down to just 4 minutes per talk on average. Even when taking on a set of like talks at once, that is remarkably fast. The real problem is that with a few minor exceptions, there were no real overlaps. For a set of like talks you really had multiple venn diagrams of coverage; which hurts my brain. All in all a truly herculean effort.

I would like to think the PyCon-Tech stuff helped things along, even with the tagging, restructuredtext, and form issues. the only thing I am certain of is that it would have been impossible to have taken in 140+ proposals via e-mail and managed the decision making process in a wiki. With that said, I would like some feedback on the software if you used it. Also check out our trac site.