I was not going to do any posts about the PyCon proposal review process until it was done, but I am sitting here with a head cold, and hopped up on med’s, and my pet-peeve button has been pressed one too many times (so we will go with that excuse, verses me just being a disagreeable, argumentative, melodramatic person 
). This does not directly relate to the proposal system, but it is a very useful data point. Obviously I do not hate tags to the exclusion of their general use. This post and all my others have tags. I have not always been so averse to tags and saw semantic tagging as being the first step into a more useful web experience. The real issue is that every one has their own concept of what a tag is and how they are used.
This topic came up on the organizers list a few weeks ago. Laura Creighton made some very good observations on how we were crossing purposes with the tags on proposals; using them for reviewers, speakers, and attendees. She notes that the tags which are useful to reviewers are not those that are useful for people wishing to attend a talk (software issues kept me from implementing this). Herein lies the first problem with tags. Each person looking at the tags will be trying to use them for a different purpose, and you can not tell that person not to use them in that way. Some people want them to filter multiple objects of some type, some want to find related information, some are more interested in the occurrence of specific tags, while others find information in the tags them selves. The holy grail is to try to have one set of tags achieve all goals, but it just can not be done, as the review process has shown.
To me, the best example of tags just not working, is the tag cloud. When I first saw a tag cloud I was in shock. “Why didn’t I think of that?” It seemed so obvious that the tag cloud, with size and color, conveyed a depth an immediacy of information that it was one of those revolutionary ideas. A revolutionary idea is one which is very simple, easy to use, and yet extremely powerful. Then I started trying to use sites which implemented tag clouds. I now despise the tag cloud, and dismiss sites which use them. Why? Because they do not provide an interface into the information I the user care about. A python based blog has a huge ‘Python’ tag and a number of other medium sized tags all related to python, and then there are a few little tiny, impossible to read tags, one of them being ’svn’ say, and that one post contains the nugget of information I care about. If there were a way to make the cloud represent the size of tags by how important I deem them to be at that time, then it would be a useful interface. As it is, it is an annoyance, and completely useless as a means to find information, which is presumably the purpose of the interface. the one place where I thought it would be usefu, news, where the size is representative of the number of news articles of the day or week which have that tag, falls flat in my experience as a user. I rarely care about the most talked about subjects of the day. I care about the little things, the oddities, the things that matter to me. I guess what is important to me is not what is important to the majority of other people. Maybe if I cared about what everyone else seems to care about (if these clouds are in fact representative of that), then they would be useful. But are they even a good measure? How would one determine that?
This is the general problem with tags at the end of the day. They are representative of what the person assigning the tags feels is important, not the person who is using the tags feels is important. So why do I use tags on my site? They are for me. I really do not expect them to be useful to anyone else. The tags help me find stuff I have posted previously, and to generate some tag specific feeds. This post will not show up on the unofficial planet python, as it is not python related. In that regard the tag is a content filter for that aggrigator, which again is more a use for me, than anything else.
So what does this have to to with the pycon proposal system? Well last year when I first wrote the proposal system I included a free form text field for people to add categories. No one could agree on how this field would be used, so I just left it like that and figured the program committee chair would figure out how it would be used. Keep it simple, and let procedure determine the rest, not code. Its a meme I use often, and it provides for great flexibility with little (usually no) code work. The proposal system was in a lot of flux at the time, and we never did come up with instructions for that field. The end result was a haphazard representation of like categories with little overlap. There was ‘web’, ‘web service’, ‘web-interface’, ‘web framework’, etc. Jeff Rush spent hours going over the list and standardizing it to what we ended up with. It is a fairly long list, but not too long for the number of talks, and it fits on one page on all but the most restrictive of resolutions. We did not want a repeat of this and when I asked for feedback on the proposal system, the only specific feedback I received was on the categories. What people wanted was:
- Have example categories to choose from.
- Allow for people to add their own.
- Have those added categories in the selection.
- Allow for spaces. (At first I liked this, but now I regret it)
- Limit people to 10 categories. (I wanted 3 and should have stuck to this point)
- Do something better than the usual multi-select as the control-click thing can be a pain.
I would have ignored that last one, except 5 out of 8 pieces of feedback mentioned that. So I cheated and used the django multi-select widget, and seeded the tags with ones from the previous year. There are some public examples where you can see the end result. You will also see all the tags people have added over the review process. the hope was that as people would only add a few tags as needed and would reuse the ones already present. The nice ’search’ feature on the widget makes things even easier with limited form real estate. The end result is we have 113 unique tags being used for 141 talks! We have 6 tags which start with ‘web’, for which ‘web’ would be perfectly fine. We had 5 talks which used the tag ‘develop 3x faster than others’?!?! The truly humerus part is that two of those talks were claiming to be faster/better than the other. I have removed that tag, and will be going through the others as well. One person repeated the title of the talk as a category. There are others whom have created lengthy categories resulting in a category list which is longer than their talk summary! The only thing I can think of to explain all this is that the form interface still sucks, and the process is still not described clearly.
I will be going through all the proposals and doing a category house cleaning, and try to get a handle on things. I think I will limit things to about 20 categories, and next year those will be the only categories we will have. We will also limit the number of categories per proposal to be 4. Hopefully this will make categories have some actual use. Currently they are nothing more than an annoyance, and have little value.
I should note that I am not a web designer, user interface specialist, or web anything. I am a C++ developer with extensive python C/AP and integration experience. This PyCon-Tech stuff is a lark, and 100% out of my comfort zone. The things I want to work on (survey data mining using NLP, interest statistical analysis, group theory for google maps), I have had to put off as more important core features are needed. Think I am completely off base? Think you can do better? See something about the site you don’t like and think it sucks? You are most likely correct, and no I do not know how to fix it. So PLEASE step forward and help out!
I have some sympathy for you, Doug! Proposal review systems seem to bring out the bikeshed painters, leaving a potentially huge amount of work to be done whilst never quite satisfying any particular group of users. And it’s arguably a classic case of one group being the “stakeholders” (the reviewers) with lots of input on how the system is supposed to work, tuned to their workflow (of course), and another group being the people who complain that it’s unusable and not suited to them (the submitters).
I don’t think next year’s EuroPython has settled on anything for this purpose, but I doubt that I’ll have anything to do with this part of the organisation, for the reasons given above, amongst others. Let the reviewers build their own system!
Extra sympathy for you, though, if you are a reviewer for PyCon.
It’s interesting to hear about Laura’s observations about tags and your complaints about the misuse of them, but I think the issue is really one facet of a wider problem: namely, the focus on technology to solve what is basically a social problem.
For example, it’s terrible to read how Jeff had to try and organise tags. It seems that introducing a tool to solve a problem just ended up causing more work and actually distanced the organisers from the problem they were trying to solve.
Finally, it’s unfortunate that you are in the position of implementing something that you don’t feel qualified to do. This leads on to some observations of my own…
At EuroPython this year, the people involved in parts of the organisation of the event openly complained about the amount of work that needs to be done to make a conference happen - I think it was a useful thing to do to make people aware of that. One result was that people stepped forward to offer help with the website, organisation process, scheduling, and so on. Now, I’m not sure that this has necessarily translated into action, but I think that may be because people need some kind of project management or supervision, and perhaps you should be performing that role rather than actually having to work on building things you don’t have much experience with.
I realise that, in the end, if there’s a shortage of volunteers then somebody has to get their hands dirty and just do the job at hand. It’s always a shame when things end up that way.
While I don’t disagree about tags being used for different purposes, I do disagree with your assessment on what the purpose of a tag cloud is. I have always thought of a tag cloud as a way of seeing that the “hot” topic is in a web community. As long as that community isn’t specialized to the point of having one topic being used in every post, and hopefully many people contributing, then the tag cloud can be a very useful way to navigate what is important to the community.
A tag cloud on a blog with a single author or with a highly specific purpose and then using that as a tag can definitely defeat the purpose of the tag cloud.
I do like your idea of providing a limited set of categories and allowing people to choose a finite set of them and I have also come to accept the idea of not allowing spaces.
Paul,
EuroPython is looking at PyCon-Tech as one potential solution. PyconUK used it this summer, and 3 other conferences are interested in it. As for letting the reviewers build their own system, they have a hard enough job without having to do coding. That is one of the reasons why I took on that part; I had experience with StoneKeep Software’s solution, and really liked it. I wanted something similar for community conferences.
David,
I agree it is a social problem, but we need something. I just don’t know what that something is anymore.
While I do not feel 100% qualified to implement the solution, from what I have seen, I think I am doing a decent job (at least I hope I am). I don’t mind working out of my comfort zone; I just wish I had more help. Someone with UI and design experience would be of great help.
Brantley Harris did the new design for the website and it looks fantastic! the problem is he has been extremely busy and not had much time to devote to it. I used the django admin CSS for much of the interfaces (because it is what I had previously), and now the two designs are colliding in bad ways. Brantly just did a one page static mockup which I templatized.
I am doing many things wrong with running an open source project when it comes to getting volunteers, but that is a different post. That combined with the fact that conference software is just NOT sexy means not that much interest in working on it. The code is very complicated which appears to be a barrier for some people.
PyCon-Tech is now actually 9 full featured django applications which can be used separately or integrated together, with 3 more big ones comming. That can be daunting.
Jerry,
I agree that the tag cloud would be useful if I actually gave one fig for what the hot topic was in anything. So far I have been severely disappointed in the blogosphere (or whatever) and every news source. I just don’t care about what all these other people care about. Not once has a hot topic interested me. I might be getting old and out of touch, but I think it has always been this way. Even in the middle of the Harry Potter mania, I was completely uninterested in 99% of the Harry Potter anything out there.
So to date, the clouds have been the worst interface for me personally, and as such I want nothing to do with them. If I find one aggrigator which I find useful, I might change my mind, but after 4+ years of the cloud, I am not holding my breath.
Sorry if you thought I was criticising your project management skills - that wasn’t my intention. My point was that, ideally, the tasks would be done by the people most qualified to do them, and there would be the time and resources to do the things you really want to do. Now, we all know that that isn’t the way things work in real life…
It’s now too late to change course for PyCon 2008, but it’s probably still worth throwing the Wiki page into the mix to help focus discussion of software for later conferences. Ideally, there would be a central mailing list for this discussion - I don’t think it helps to have separate discussions on EuroPython and PyCon-specific mailing lists.
Doug, I think that there a number of factors in play.
First of all, there’s the complexity in getting started, and I think the EuroPython infrastructure has some inertia in this respect. We used Indico for 2006 and 2007, mostly because CERN developed Indico and preferred to use it when they were running things in 2006, and we only continued to use it with CERN’s support in 2007 because no-one was inclined to deploy anything else. Indeed, we didn’t even get as far as to deploy Indico anywhere else, and that really caused inconvenience for users later on. But it really takes people from the target audience to get their hands dirty, which doesn’t always seem to happen.
Then, there are the perceptions of the target audience for whatever does get used. Indico makes itself a convenient target for criticism because it’s meant to be able to do almost everything, so if the reviewers don’t want to use it for talk selection and scheduling, preferring someone’s kitchen table instead (which is up to them, of course), then Indico gets a reputation for being inflexible or whatever. If users find the registration too confusing, it’s because Indico’s registration form isn’t shiny enough, and “why didn’t you do a Django application with AJAX?” (As if I had loads of time to burn and decided to spend it playing with Indico.)
In my experience, as the scope of a tool or set of tools increases, the number of user types who have to use it also increases and, inevitably, there are then fewer users who have good things to say about the experience. That’s not necessarily the fault of something like Indico, however: as David says, there are lots of social aspects involved in organising a conference, and it seems too easy to me to just blame the tools when humans were responsible for choosing them in the first place.
So, I’ve come to believe that each special interest group is best served by choosing their own favourite tool, even if the infrastructure might be more complicated as a result. I don’t care how the reviewers choose the talks (and was actually more concerned about there being enough talk submissions in the first place for EuroPython, which is another story), and if they don’t manage to arrange a nice Web-based solution and end up doing it by having proposals arrive by e-mail which they then print out and pin up on a noticeboard, or whatever, then as long as it doesn’t impact the registration system and the general publicity, then so be it.
Especially in the light of “democratizing” conferences, the “high volume” participant-facing technology is the most important part before the event itself: information about the conference, where/when it is, who it’s for, how to register, where to stay, who else is going. I’d go as far to say that all other “virtual” processes are “luxury processes”, but I’ve probably left something out of that list. Sadly, when I’ve read anything about conference technology development, the people most excited about it seem to be more interested in the exotic stuff: geotagging, Web 2.0, social networking, tag clouds
and so on. My embittered personal feeling about this is that it’s like designing the ultimate spacestation before inventing the rocket (ie. getting people to register in the first place).
Perhaps there should be more experience sharing between the conferences, though. It needn’t be all horror stories and complaints.
I think limiting the tags is good. I really like that Google Code Hosting lets the project owners sets what tags are allowed for people to work with. Keeps the explosion of tags from occurring.
And from a reviewer’s perspective, I have just ignored the tags. =)
David,
Oh, there is no question that my management skills for PyCon-Tech thus far suck eggs :-). I know what I need to do to fix things, but life keeps getting in the way and I just don’t have the time. I am hopping that as more clients come on board (there are 3 other conferences looking at it), that I will get my butt in gear. I was even considering giving a talk ‘How not to run an open source project’, but there is still a question of wether I can make it to PyCon. If it happens it will be an open space talk.
Paul,
Thanks for the history behind the EuroPython efforts. I completly agree, you get out of the system what you put into it. I have not used Indico myself, but I have heard comments on both ends of the spectrum about it. I have also heard the exact same comments about PyCon-Tech. I even had Jim Fullton yelling at me durring the sprints about the software :-). (I was not upset at all, he was right to be mad, and all I could think was ‘Wow! Jim fullton is yelling at me!’)
I also agree that the reviewers need to use something that they are comfortable with. I fear that parts of the proposal submission software have become a barrier to submitting talks (even with the 140+ submissions). That is the worst possible outcome for a submission system, and one that I hope to get some good feedback on.
I have also been talking with someone from a rather large international conference about forming a more centralized place for discussing conference management and technology. Hopefully something will come of that.
David,
I know about the requirements page and have withheld from commenting. I am also on the europython-improve mailing list. The requirements look almost identical to the ones that AMK wrote up in 2004 for PyCon, and were the basis of PyCon-Tech (origionally his project). They are missing way to much to be considered real requirements. Registration and Peer reviewed proposals (missing in the requirements!) will be the hard parts for PyCon-Tech.
At the end of the day registration is the #1 thing to get working well. Everything else can be done with e-mail and a wiki. This does not scale well, and I am very glad to have a software solution in place for PyCon 2008. I do not believe we could handle the load otherwise.
Brett,
I should ignore them as well, but find myself unable to do so ;-).
Hi Doug,
You already know my reStInPeace soft, maybe you can give PyK! a try as it is an evolution from the first one and can help you writting good articles for PyCon 2008 in reSt format for example.
You’ll find it here, and of course it’s free !
http://kib2.free.fr/PyK/
Cheers,
[...] big fan of their tag cloud comparsion tool. But tag clouds don’t have the best reputation. Some people outright hate them. And the people at Many Eyes admit they’re a bit of a “toy”. [...]