Dougma (dŭg·mə) n.

Well no bears this time, and no pictures yet either. We had a fun time at our annual first camping of the year, and Sarah had a blast. The weather was perfect. Everything went great. Josh was a great big brother showing Sarah the ropes and introducing her to everyone. One of the highlights was playing tag with Josh, his friend Quit and my friends Matt and Deidra.

But for some reason I just did not relax. I didn’t get to spend as much time with friends or just vegging on the beach as I wanted, and I really have no one to blame but myself. Part of the problem is that I just could not shut my brain off. Last year I got a notebook, and it ended up being my ‘PyCon’ notebook. This year, I just didn’t seem to have time, oddly enough. Code freeze at work was Friday and that, I am sure, didn’t help. The official fork will most likely be this upcoming Friday.

I have had no time to work on any of my python projects, and it is driving me crazy. There are so many fantastic things happening with django-survey, and Pinax is picking up steam, I need to put the old PyCon ‘08 stuff into archival mode, and start up ‘09. There is some very very interesting stuff going on with the DFW Python group that I want to help out on as well. There are so many fantastic things being worked on right now by incredible people, and I feel a down right claustrophobic not being able to do anything myself. I only been able to attend one Boston Python Meetup so far this year!

It looks like I will be in Montanna the first week of July this year (my anual pre-PyCon-Tech kickoff-kickoff). Barring any project ‘issues‘ I hope to get my act together then (with respect to PyCon software for 2009). If there are any pythonistas in the Missoula area, please send me an e-mail! For now I guess I should get back to unpacking the car.

Steve Holden is participating in the 5K Race for Hope. He is looking for people to sponsor his run. Lets show the other groups the generosity of the Python Community! (Sorry Team Hopkins, Steve got to me first )

If Steve is willing to go the full 5K distance, we should be able to support him with some cash. With the exception of this past year, it has been his fund raising efforts which have kept PyCon so cheap. Lets put some of those saved pennies towards a great cause!

I have been holding off on writing this post as I prefer to fully form an opinion.

At the writing of this there are almost 600 blog posts about the new google hosted application. Most seem to me to be flailing around the actual core of what this new little beastie is. Some are comparing it to Amazon offerings, some as a threat to commodity hosting, and some as the dawn of a new computing revolution. A few highly respected people see this as brining application development to the people the way that html/aol/myspace brought web development to the people. Many see this as a validation that python is an enterprise level platform. While I believe python is just that, I do not yet see this as a validation. The validation comes with Google App Engine’s success. Not that the language needs this added validation. As for the revolution, time will tell.

Google is toting this as making the web a platform. A platform for development, essentially replacing the desktop as where applications get developed and deployed. They do all the busy work of setting up the hardware, configuring systems for monitoring traffic, setting up the database, setting up the source control system, bug tracking, and all the rest, and let you focus on writing the application. Also you get the power of Googles massive data centers with literal warehouses of machines and disks and their custom database.  They bill this as being a platform for building your web based applications, a user base, business, and revenue stream. Of course one revenue stream will be ad-sense further promoting the Google advertising juggernaut. This is all fantastic, but there are limitations (as there must be). I put the limitations at the end.

What is Google really up to?

Google is not releasing the App Engine in a void. There are many other services that google has been rolling out over time (and many quite recently) which need to be looked at in order to get a proper view of what is happening. First lets step back a bit and look at Google’s past. In the past when google released a feature with an API, people would rush out and start building mashups. Mashups which combined parts of Google, and parts of other systems. Systems google often had little or no control over. Early on Google revoked some keys when things got out of hand. Very early on there was some bad PR. Some mashups went away, some came back, some just died out. There was a wealth of data, information, and potential revenue behind those mashups, but it was out of Googles hands by and large.

In the mean time Facebook came along and changed what it meant to write a web app. No longer were applications monolithic disconnected things. They were widgets which plugged into a page. They were cool integrated, socially networked, and shared. They were things people paid real money for. They were things people were using to generate adwords revenue! Google created their own apps. They created them for all the other social sites and the desktop. they did not care who was the hot new trend, as long as they had a share. But they have to play by other peoples rules and API’s.

So google has their search, mail, maps, and documents, online bookmarks, and calendar. They have an rss reader. Others are making mashups, and now many of those are occurring on FaceBook as 3rd party apps. Google releases some extensions for form filling on the docs, and integrating charts. They release a data API. They release OpenSocial as an attempt to standardize all these social networks and the core of what their apps provide; the social connections. They release custom site hosting (without announcing it except in a blog posting). They now have all these great applications and pieces of applications. They have a means of creating, editing, and hosting static html and data. What they lack is a framework to integrate everything. Something where they host the mashups. Something that they can do the deep data mining on. At least that was the case until App Engine came along.

Google claims in the very opening of their announcement that the App Engine is all about the developers. It is all about the people out there who develop neat and interesting things and the feedback loop that creates. The creative creativity of the masses. That has always been the key to Googles overall success. They provide the tools, and others create all those great mashups, sites, and apps. This is not about you creating your cool new app. This is about you creating your cool new Google mashup app utilizing all the other google API’s. They are not all there yet, but they will be. The crucial one, the user backend, is already there. All the other offerings do not require their python API. All the other offerings already have javascript and IFrame, and other means of integrating which were developed for integrating with your blog or MySpace or FaceBook. But make no mistake, they are coming to GAE.

In short this is about taking all those Google pieces parts and creating the ‘next big thing’ and using the developers out on the internet to do it, as they are the ones who will do it anyway and now they can do it for Google.  Google gets their precious data, their add revenue, and at some point people get to pay for the privilege of developing apps for them (either via adds or real money for removing those quotas).

Now comes the really cool part. The SDK includes everything you need for running locally. They have the Google Gears framework for making your apps work both online and offline on your desktop. Integrate all that fully and there really is no difference between your online web based apps and your desktop apps. There is still a long row to hoe before it gets to that point, but the pieces are falling onto place.

Why Python?

There are a number of theories about the real reasons for choosing python. Most believe it’s because python is one of Googles 4 primary languages. I do not believe that exactly. If this could have been done in Java, they would have done that. PHP is the only other ‘language’ that could approach what they want to do. As what they want is a platform for developing mashups with their existing technologies on a massively distributed scale by unknown random people, here is a short list of requirements:

• Easy to develop in (who would develop in prolog?)
• Sandboxable (including no ability to crash the server or corrupt ram)
• No spawning of processes/threads (or other things to bypass cpu/process management)
• No connections in or out of the app except those expressly controlled via an API
• Easy means of administration (for the developers)
• A language for which the Google API’s are already available
• Low overhead for deployment on the servers [initial startup cant be too slow, later requests must be extremely fast]

I know of no other language which meets these requirements. PHP comes close, but would require a partial lobotomy (where python just has some modules removed or limited). Also PHP is not one of the languages that there are API clients for. I know that people are clamoring for other languages. All I can say is don’t hold your breath. I just do not see it happening any time soon.

[UPDATE: as a commenter points out, google is quite dedicated to python and has many core programmers on staff including the language creator. This is a great help for getting things done and adding validity to the project. Read the comments to hear my thoughts on Ruby.]

Growing Pains

App Engine is in its infancy. As with all their Beta projects there are problems. The main problem is how they are dealing with the problems. In short they are overwhelmed. People are asking for PHP, and their favorite python projects to be supported. They made the mistake of claiming that most python frameworks will run on it without putting up the proper CAUTION signs. It is possible to get Zope to run on it with some work. All that is missing is the hook to use the google database instead of the ZODB as the backend, a few minor tweaks, and use the WSGI adapter. Twisted is just out due to the signals, and the threads, and crucially, the tcp connections. One of the problems is that people expect that XYZ module should just work, and it’s the App Engine teams job to do that. The team seems to feel that they provide the framework and others should do the porting. There are also reports of bugs not being responded to in a timely manner. This is a bit laughable given the shear number of bugs currently reported and the 15 or so engineers they have dealing with all the App Engine deployment issues. I am sure that no one expected to have to deal with flamewars in the bug tracker. Or that thousands of people would post +1 comments in the bug tracker making it next to unusable (some people just can’t read instructions). I would not expect all the current bugs to be triaged until late next week or the week after.

Most of the complaints seem to be about the limitations put in place. I can understand that, but I can also understand why hell will freeze over before most of them are lifted. When it comes to an initial deployment it seems quite generous and unrestricted. Insanely so. If you think about what it takes to deploy something like this, at this level, things start to click into place. How would you do it? How would you manage the issues, security risks, vectors of abuse? It is great to say you want to create thread to accept a certified https connection, but if you are making that request, then you have no clue about the technical aspects behind that request or the technical aspects behind the App Engine.

Current Limitations

1. No long running processes

These are run once executions, and there is a time limit of a few seconds. Think of this the same way you would think about a PHP page.

2. No reliable state between runs

There is potential state from one run to the next, but you should not rely on it for large deployments. All state and persistent data should be stored in the database (or via some neat hacks). NOTE: this is more from my reading between the lines and knowledge of load balanced grid deployments. I.e. I do not trust their ‘cache‘ system as something that can be relied upon.Why? Because we are talking about nodes and sandboxes.

3. No incoming TCP connections

No binding to sockets, etc. These are Google’s servers. Even they do not know which node your http request which starts the app will be run on; no way of knowing which IP it really will be. Only apps are running on these nodes. This means no mixing of non-app and app requests. No twisted or zope admin instances. For google to provide a proper balanced network (with proper dispatching), it has to be that way (well at this phase in the game at least).

4. Limited connections out

Google has a url API for making http and https requests out to other servers, a connection to a database and a mail API. Those are the only outgoing network connections, and all are bound in API’s. If you were allowing anyone to run programs on your servers would you want them to be part of botnets?

5. No https

This is not static IP hosting, no cert for you! There are some things that can be done, but there would be cert warnings, etc. Granted this does not stop you from integrating with PayPal, or Google Checkout, where the https checkout is handled by a different site (insanely weak). [UPDATE: yes I know static IP's are not required for certs, but they are required if you would rather people not to get the cert warning or have IE7 mark the site as 'insecure'. And google will not pay for a cert per app, nor will it get a single cert for all apps some of which they are not really sure of their authenticity (a phising app based on Adrian's dynamic html->template tool for instance.) I do expect them to support something in the future, but that is a ways off and will not be for free.]

6.  No spawning new processes (or signal overriding)

Well no big surprise here. Starting new processes could be very dangerous for all, and signal overriding… well that could make it hard for google to safely stop a rogue app (among other things).

7. No creating new threads!

Ok, this is a bit strange at first blush, but if you have ever dealt with grid deployments, or taught a 102 CS course (where you start covering semaphores and mutexes, and IPC) you have had the experience of a rogue multi-thread app taking down a  machine. Part of the problem is that creating a new thread is very much like starting a new process. One of the interesting things about starting a new process is that the operation does not adhere to the nice protocol. It gets the CPU to do that start no matter what, and at the kernel priority (which is not very nice). New threads behave the same way, and are a PITA to deal with when trying to take down a process which has gone rogue. I hated that lab. I have other theories behind why they do not allow this (but that is for another post)

8. No ‘real’ filesystem

Well there is no real access to the file system. Not the ‘real’ file system. As such certain things like tmpfile are not present (as there is no /tmp directory).

9. Crippled import/bytecode

Well that is an overstatement. Google has written their own import replacement, and modified the bytecode (I think) from standard python. I have some theories on why for a later post, but the deal is, forget about using marshal, imp, or even some of the package __import__ hooks, and cPickle is just pickle.  Part of the reason is because of the lack of a ‘real’ file system. The python path and import control is special as only packages from google, and those in your current app are available, and they are specially managed. This should not affect anyone unless you play funky package import tricks that you should not be doing anyway. Extending __path__ in packages does still work, but using __import__ directly to import a package using a computed abs path does not work (might be a bug).

10. Quotas and App shutdown

If your app gets too popular and goes over quotas, then it is disabled. Once it gets too popular,  you need to buy more computes, etc. None of the quotas are set in stone yet, and of course if you use google analytics and/or ad-sense, then the quotas are less restrictive or removed. The details are still in flux. For the beta period you can request a larger quote for free (but each request is reviewed for merit). You can also report app abuse if you find that someone’s app is not being nice.

[UPDATE: Here is the link to the current quota system.]

11. Only 3 Apps and no deleting.

For the beta period, each developer can have just 3 apps, and you can not delete an app.

12. Only pure python

No c extension modules. This is again because of the sandbox system, and all the other stuff above. You can’t prevent process or thread spawning in a c extension. You can not stop a c extension from corrupting things in very bad ways. You can’t stop it from attempting to connect out or bind a tcp port. And it would be a PITA to distribute the binaries to all the nodes like they do for the apps themselves (via custom import hooks + caching).

[UPDATE: fixing numbering and adding some other restrictions and errors people have pointed out]

13.  1MB per file upload limit and 500MB total storage limit.

The 500 MB limit is part of the current Quota system, but I was unaware of the 1MB file upload limit that a commenter pointed out.

14. 1000 files in an app limit

This is a huge problem for people trying to deploy pylons, TG, or Django trunk based applications. One potential solution (which is not currently supported) would be for google to allow for python zip imports and have things bundled.

15. The Google DataStore has limitations over a classic RDBMS

Ben Bangert has a great write up on this, so go read that.

Well, this was going to be three or four posts, but thanks to some interesting announcement from google, it all sort of runs together.

It still will be I think. I will most likely try to rewrite things to give an overview and go into detail on specifics later. Things are getting interesting at work so we will see how much time I have to pull that off.

Files

Ivan beat me to the punch on the main gist post. While at PyCon I had the opportunity to chat with Mike Fletcher, another OLPC volunteer whom I forget their name, Phil Hassey, Richard Jones, Jeff Rush, and about 5 other people who wandered in and out of the small sprint room we were all half passed out in. People came and went durring the discussion (I believe Richard and Phil went off to play a board game at one point as well) which ranged from modern Sci-Fi offerings to games to global warming being a net win for Canada to the history of the world (not the movie). I should have gone to bed well before the discussion started. The discussion turned to the object store on the OLPC platform. Jeff, coming from a ZODB background, was quite pro object store systems replacing ‘file systems’. This is a hot button topic with me. This topic has come up at every professional job I have had going all the way back to when I was an CO-OP at Motorola as a ‘Document Administrator’ (secretary). In fact the only two topics which are more hot button for me are ‘common application UI framework’s, and ‘security after the fact‘. I first started thinking about this subject back in 93 when I first started working on  MUDDs (warcraft, only 100% text for you youngins). The world was editable online (like a lisp MUSH) but also had revision history (via RCS initially). We were dealing with ’serialization’ and how objects were managed. I fell in love with the idea that everything could be described as having a set of attributes (tags) and really you wanted to store and manage these things by those attributes. Permissions were nothing more than attributes. Actions were nothing more than attributes. Meta data by definition were just attributes. We struggled with systems for this, but I came away convinced that we needed a new paradigm in object storage, and this ‘file’ stuff was running on borrowed time. It came up again at Motorola for document management. It came up again at OpenVision (later Veritas) for backup and security compliance. It came up again with ClearCase and Derived Objects. It came up again with ‘dictionaries’ and data management for VoiceXpress. And the code base I currently work on has something called ‘DFiles’ which I can not discuss except to mention the name (DRAT!)

Storage

Back to the discussion at PyCon. I wish I had a transcript of the discussion (no I don’t… I was not as coherent as I think I was). The Idea that everything is just blobs in a cloud of data where the tags determine the meta-structured is nice, but there are some problems. The first and most obvious problem is that it does not integrate well with existing technology and libraries. Decades of software has been written with the concept of files. You can try a fake mapping, and try to integrate things, but it does not work well. Then there is the concept of ’sub-blobs’. That is each of the pieces of data could have sub parts. This maps well to your document which might have a chart or spreadsheet as part of it for instance. This can greatly simplify serialization, and you get all those nice blob store things. Your in-memory structure is your serialization structure. But in reality we already have this. They are called files and directories. It is simply (*cough*) an implementation detail dealing with the storage mapping. Ok, there is nothing simple about it, but we will come back to this. The argument then turned to the fact that you can’t have a blob show up in more than one directory. False. Those are called symlinks, but again that is an implementation detail. One of the biggest benefits of an object-store-as-filesystem is the ability to find and manage things not in a ridged tree structure which does not scale well in the average human brain (where did I put my (ssh) keys again?) But in practice it is just replacing one confusing arbitrary structure with another on some level as it’s usefulness is measured by the quality of the tags, attributes, and indelibility of the data.If you had those things well defined in a directory tree structure, then it works just as well (as google desktop search proves). A more subtile problem is that not all tags/attributes are created equal. It took a long time for my betters and practical experience to prove this to me. Many attributes are only useful to programs. These programatic tags are for relating data, validation, encoding, and the like. Most of the time these are auto generated or involve mathematical computations. They are never intended for human interpretation,but are none the less crucial for data management. You can try to predetermine the different types of these meta attributes, or just lump them together, but neither of these approaches are really tractable. Spend some time deep diving into the abuses of the windows registry and you begin to get an idea of the issues.

I know I am glossing over all the details, and not really giving any points the attention they deserve. I am not even properly quantifying the points. Issues of language are completly being skipped over (try describing what a ‘word’ is in your application; try again when that application deals with speech and natural language… how does that abstract into meaningful tags?) Oh well. The point is there must be a happy median. We should be able to have something which has a file system programatic interface, as well as a generic data store interface. The browsing of the data should be an abstraction. If this is implemented with a classic journaling file system or in a database should be an implementation detail at the filesystem level. Why invent a new abstraction layer which everyone must now implement against when we have a perfectly good one that everyone already does? A file by any other name still contains data. If this is such a good model, think about extending it to namespaces. The problems in software code management (which is just data on a very real level) for which namespaces were invented exist on the filesystem as well. Chew on that while you code with Matrix.Optimizer and Optimizer.Matrix.

Google

Google has an interesting take on all of this. All of their service (news, documents, reader, calendar, mail, blogger, etc) all have a file like data storage for the objects represented. They use folders/directories (really tags). The only restriction is that the folders are only one level deep. I do not care for this myself. I would love to be able to have a ‘people’ folder under my ‘python’ folder and have only those times tagged with both ‘python’ and ‘people’ under that ‘folder’. Maybe that is just me. I would not want these sub folder relations to be automatic. I would want control over the layout, but have the population automatic. But that is the only extension to their system I would like to see. Beyond that it just works. It works with both the object store model and the file/directory model. If only google would open up their API’s a bit more to include this system. On wait, they just did. You know if I had hit ‘publish’ on this post last evening when I first wrote most of this, I would have been ‘prophetic’ or at least ‘first post!’.

It’s not all hearts and ponies and sparkle (even if it is python and an abstraction layer on top of django to boot!!!) I have been holding off on posting this err… post until I could formulate a non-reactionary opinion on the entire Google Apps thing. I now have an opinion and it is much along the same lines as Duncan McGreggor. The issues I have are both similar and yet unique to his, and I will post on them separately.

Python, and Conference Software

This post is already too long,and my laptop battery is dying (no the charger is at work ). Those of you that I talked with at Pycon about UnConference hosting know what this is all about, and I told you so ;-). The last piece just fell into place. With that, good night

And visa, versa.

I had a wonderful talk with Michael Trier and Brian Rosner who do the ‘This Week in Django‘ podcast. Lots of useful information, check it out. (And you get to hear my voice….)

UPDATE: Here is the image I mentioned in the podcast about [name withheld] watching a presentation Guido gave at google, on google video, at PyCon 2007, while Guido was giving a talk. Thanks again to Sean of Tummy.com for doing the networking last year, and making so that people could do things like this and not have it effect the network!

First off, early-bird registration ends in less than a week. This year has a fantastic lineup, and over twice the tutorials of any previous year. Not to mention the sprints which are free to the public.

We have a new registration system this year. It has its quirks, and there have been some issues. Most of these were expected and we had plans in place to deal with them. AMK has taken on the role of the Registration Manager. With things in his more than capable hands everything is running quite smoothly. I can’t help but stick my nose back in to resolve some issues. To be honest I do not have the temperament (or time) to deal with talking to actual people (code is more my thing). But some people I already know, or requests were made and I knew the person was still online. AMK didn’t help things by documenting some standard responses to some of the common problems, which made it too easy for me to ‘help out’ ;-). As of this writing we have more registrations than we had last year at the close of the early-bird period (375 in 07, and 390 right now).

So why do I have the Blues? I can’t decide which tutorials to take. Work is paying for me to go this year (and has even ponied up for a sponsorship!) That means that they have a say. This is additionally complicated by the fact that I still don’t have my XO, and it’s questionable that I will get it in time. What this means is I am waffling on whether to take ‘SWIG Master Class (David Beazley)’ or ‘Making Small Software for Small People, Sugar/OLPC Coding by Example (Mike C. Fletcher)’. I really want to take the Sugar class. We use SWIG at work, and not the simple stuff either. We don’t do any of the C++ stuff, but have things like type safety on our constants with custom repr’s and the ability to do things like pass in an array of length 0 and say its of length 100; have to test those memory error conditions after all. Callback functions are supported, oh and it’s thread safe, with all calls releasing the GIL safely. SWIG gets really painful at that level.

In the afternoon it looks like ‘Tools for Scientific Computing in Python (Travis Oliphant and Eric Jones)’ for me. There were hard choices here, I want to learn WxPython, and the generator tricks look very interesting, but in the end SciPy is the best choice both for work and for fun. With my existing experience I feel safe in skipping the morning companion tutorial. We use the SciPy packages including numpy and matplotlib quite a bit at work but my experience with the later is only cursory. I hate to admit it, but many times I use win32all to push the data into excel and generate graphs that way. I have it on my list to write an FFT and a group theory approach to pitch detection using SciPy; yes that is ‘fun’.

The evening is the toughest. This is where PyCon-Tech related tutorials come to the fore. It is down to ‘Practical Applications of Agile (Web) Testing Tools (C. Titus Brown and Grig Gheorghiu)’ and ‘Django Code Lab (Jacob Kaplan-Moss, Adrian Holovaty and James Bennett)’. I would love to go to the code lab and dive into parts of PyCon-Tech. The hard part would be selecting something small enough to discuss. PyCon-Tech has grown quite large and some of the parts are quite involved. Just giving an overview of the registration system can take 3 hours (from actual experience). There are parts of the proposal system which need optimizing (but again explaining the issues alone could take too long). The real problem I would love help with is the separation of the display (html templates) from the core of what I now think of as the PyCon-Tech framework. We have a new design, but once again the design is coupled with the implementation. Swapping out another design is easier than last year, but harder than it was just two months ago. Then there is the 10K lb. Gorilla in the corner. PyCon-Tech has 0 automated testing. Guess I will have to go for the testing tools, and hope to snare the Django folks some other time. To be honest I feel weird with the Idea of bringing PyCon-Tech stuff to a tutorial; as if I am stealing time from attendees for conference related stuff.

At some point I need to do a post on the registration system… The idea was to keep it simple. Just a minor update to the old cgi form with the data stored in a database instead of a text file. Three hour chat to give an overview…. For the masochistic, you can read the comments in the source code here and here (which are incomplete) and some other notes here and here (a little out of date).

After finally being done with the talk proposals on Sunday, I decided to take a break and read some of those python blog thingies to see what the clean people are up to. I was hoping to find a few topics of minutia I could loose myself in. What I found were a ton of posts on the new Google Chart API. Now I have been looking for a good chart solution for quite some time. I even discussed certain options over at Marty Alchin’s blog. I have looked at DojoX Charts, and Open Flash Chart, and many others. Dojo has the best charts, but the API is a PITA to figure out. the Doc is mostly auto generated from the code, and the samples are next to useless. I know I can do great things with it, but I recently spent 3 hours on it and got nowhere. Open Flash Chart got me up and running in no time, and has some nice python bindings, but its a flash based solution. So when I saw the Google API, I just dove into it’s doc to see what is up, and forgot all about the blogs. Lets see what happened next shall we?…

Read the rest of this entry »

Ever wonder what the heck is going on in your django cache? Ever wish you could clear an entry or two or all? Well I have, and CacheMgr is my answer:

This does not negate the need for proper cache use, but it can be a life saver when you want to update your sites templates, but don’t want to restart the server because people are say, using it to submit proposals before a deadline. This app is not feature complete. Currently only the following cache backends are supported:

• dummy
• simple
• localmem

The remaining backends need to be implemented. The database and file based backends are simple to do, but memcached will take a bit more work. The system is extensible, so you can extend it to work with your own cache backends the same way you can with django’s cache system. All you have to do is extend the base class the same way you would with the django cache system:

class BaseCacheMgr(object):
    """No description has been set.
    """
    has_cull = False
    scheme = 'unknown'

    def __init__(self, cache, host, params):
        self.cache = cache
    def __iter__(self):
        """Should be overridden to iterate over the cache and return
        a dict with the form {'key': key, 'short_key': short_key,
                                     'repr': cache_value,
                                     'expires': datetime_expires, 'expired': False}
        """
        raise NotImplementedError
    def info(self):
        default = (str(self.cache.default_timeout)
                   if hasattr(self.cache, 'default_timeout') else 'None')
        return [{'name': 'Scheme', 'value': self.scheme},
                {'name': 'Description', 'value': self.__doc__},
                {'name': 'Default Timeout', 'value': default},
                {'name': 'Has Cull', 'value': repr(self.has_cull)}]
    def clear(self):
        raise NotImplementedError
    def cull(self):
        if not self.has_cull: return
        raise NotImplementedError
    def delete(self, key):
        self.cache.delete(key)

Fairly self explanatory when you look at the image above. Take a look at the simple backend implementation for more details.

Other features that need to be implemented:

• Clear Expired Button
• Paged View (like the admin)
• Sort Table heading links (like the admin)
• Search Fields (like the admin)
• Move the repr/short_key/expires helper code into the base class.

PyCon-Tech (the python behind pycon) is an open source project for the management of community run conferences. It is actually a framework for developing conference websites, and doing collaborative management, written on top of the django web framework. The project is broken into multiple application, most of which can be used independently. The issue is, not many people even know this resource exists.

One of the stated goals of PyCon-Tech is to give back to the python community which makes the conference possible. This series is my attempt to shine a light on some of the general use applications under the hood and how you can use them for other projects. The first app in this series is the feedutil, a lightweight generic RSS/Atom feed pull. (it is also the only app in PyCon-Tech being used for other projects that I know of).

Overview

Feedutil is a lightweight app for pulling RSS/Atom feeds onto your site with django template tags, or custom views. This is not a full feed aggregator like feedjack, but you could write one with it. This is more along the lines of the blogger plugin which allows you to have the latest 5 entries from an RSS feed appear on your sidebar. We use it on the PyCon website for the main about page which has summaries of the latest PyCon Blog entries via Atom, and on an organizer page which replicates a Trac RSS issue feed for open website bugs. This does not use any django models, and there is no database interaction. You could create your own models for managing your feeds, but that is not the purpose of feedutil.

The feedutil provides two primary template tags {% feed feed_url [posts_to_show] [cache_expires] %} and {% get_feed feed_url [posts_to_show] [cache_expires] as var %}. There is also a higher level interface to feedparser which includes caching pull_feed(feed_url, posts_to_show=None, cache_expires=None) => posts_dict.

Read the rest of this entry »

Well I have been rather busy lately and have not had time to post about anything. There has been a lot going on with ODF, and Py3K, but I have nothing of value to add on those fronts. There is a lot going on with PyCon 2008 Chicago, and I have been neck deep in Django for about six weeks now, but that will wait for another day. The big thing I want to mention is the Django World Wide Sprint! I have been on the fence about attending this as I am already overwhelmed with work, and this weekend is going to be particularly busy with stART on the Street. But the draw is proving too much.

So I will be taking part in the sprint. I have no axes to grind, so I will most likely focus on bug reports. I live in western Mass, and work in the Burlington area, so my ‘Boston’ location is a bit disingenuous. I know there are a number of Django people in the Mass area, and I would love to meet up somewhere to coordinate. My current plan is to commute into Burlington and camp out at Borders at WaySide Commons (google map). I picked this location for many reasons, but am not committed. (Reasons include: away from home, coffee, free internet, coffee, public transportation, coffee, across the street from work, coffee.) So if anyone is interested in meeting up, let me know!