Dougma (dŭg·mə) n.

I will be in Montana again in early July. This year I will have a rental car and thus more freedom than in the past. Are there any pythonistas in the Missoula/Lolo area? Or coding in general? Security?

It has become tradition that I start getting my personal game plan for PyCon together. I am hoping to do less coding and more organizational work. It would be nice to talk to anyone who has been to a recent conferences (technical or not, professional or not, python or not, does not matter).

Steve Ballmers Laptop.

Hmm…. I know I should be more concerned about what this says about my internet useage:

You have exceeded your 1027.63 gigabyte download limit
You have downloaded a total of 1027.67 gigabytes.

It’s like someone peered into my 3am brain and drew what they saw.

Randall: you forgot the farside chicken perched on the tree, but besides that, spot on! Spot On!!

Because all the cool kids are doing it:

history|awk ‘{a[$2]++ } END{for(i in a){print a[i] ” ” i}}’ |sort -rn|head

New laptop:

44 ls
37 cd
18 svn
16 netstat
13 ping
9 ssh
6 nslookup
4 python
4 mv
4 man

PyCon Server:

314 cd
82 ps
78 ls
68 tail
67 svn
59 exit
52 grep
40 uptime
38 touch
32 python2.5

Work (this can’t be right, my .cshrc must be set to keep everything for ever or 100K entries or something):

1034 pmrec
932 python
720 cd
621 gmake
612 ls
501 grep
314 tail
280 cut
210 perl
207 head

Well I think I disproved #8 but proved #9 with my last post. Writing something even if it is not ‘perfect’ is not always the best option, when what you do write is drivel.

I started to write a response to Eric’s comment and realized it was really a separate blog post, so here it is.

Eric gets a bit confused about my take on ‘tags’ and ‘directories’ and how I would like to have sub-folders on google apps where I have django/people as an example. This is because my explination of what I was trying to describe was utter crap. I had a concrete example from working on the Memorize activity on the OLPC, but I left that part out for reasons I wont go into. I think a concrete example is needed however, as these meta concepts are too vague unless you nail them down. So with that said, I will use the google RSS reader as the application in question which requires an object store.

I have been using the using the google reader app for about a week now, and I must say it annoys me a little. So you have your RSS feeds and you can stick them into ‘folders’, which are more like tags. A feed can be in multiple ‘folders’ at the same time. When you select a folder you see all the entries from all the feeds in that folder. You cna also select the individual feeds. The low level objects are the individual posts, and life at first is good. This system maps quite nicely onto the concept of an object store. Really the ‘folders’ are nothing but a semantic term for a tag or meta-grouping of the feeds and the objects in those feeds. The feed names are nothing more than meta-information on the posts as well. You coould imagine this as the ideal example of using an object store over using a file system. But as we will see there are problems.

No Sub Folders

There are no sub-folders. Why would I want sub-folders? (Yes the below are all the same problem to some extent, but manifested as different symptoms; but I will connect them up later)

1. Only a single level of organization imposed on me
Many systems limit the top level tags shown to those with more than N items in them, and then you cross link/relate from there. Others use tag clouds. Neither really works for me as those are automated systems imposing their order on me, not the other way around.2. Single level organization becomes unmanageable.

Lets look at this blog as an example of this. Look at the left column at the tags on my posts. I rarely post, and yet look at that list! Imagine managing a full machine, OS, data blobs, etc  like that! The truth is I already have 20 folders in GMail, and 30 in google reader. I have feed and mails being sent to multiple folders (tags) and after a while I can’t find anything!

3. No  means of customizing the organization

I really want to have a sub folder python/people. This would only contain the entries that  are tagged both python and people. The top level ‘folder’ people would contain all the feeds tagged as ‘people’, but would not necessarily contain a sub-folder ‘people’. I want to be the one to determine the breakdown. I want to impost the order I want on the system, not the other way around. This organization is for my benefit, and I want control over its customization. The drill down of a folder tree makes perfect logical sense. This does not preclude a system for selecting unions, exclusion, and the like in a more advanced (complicated) interface, but you need something which scales. A flat single level does not scale.

Limited Modality

One of the great promises of the meta-tagged object stores is the ability to magicly use this data for cool new represenations and visualizations of that data. Teh googel reader has some of that. You can select to only see unread entries, allow for list, preview, or full view. There are even statistics! You can see ’stared’ entries. Sort by date etc. But the grander promise is still unfulfilled.

1. No calendar integration

These things have dates and times on them. google has fantastic falendar widgets. It should be automatic right? There should be no code to write, just something to plug in and have it ‘just work’. So where is it?

2. No custom meta data

We have staring, and marking for ’sharing’ but those things and folders are meta-data which is forced on me. I want to add personal notes to myself on posts or feeds. I want to ad my own flag types ‘like I commented on this’, and ‘needs followup’; sometimes a star is nothing more than a piece of twine on your finger and just does not cut it. Object stores should just provide these things and somehow understand them for cool new visualizations if they are of a predefined meta type…. New interesting statistics should be possible on my personal meta data and the graphs should be automatic.

3. No meta-discovery

We are bringing in this data from external sources and it is already tagged with plenty of meta data. I should have access to that as well.

I am being unfair

The google reader is an incredible system, and you really do not want it to be this huge complex system. It should be simple and do one thing well, allow you to read feed entries you care about. But really I am using it (unfairly) as a prototype for a theoretical object store system. So lets talk about that for a bit. With that said I would love to have a few months working on the reader app to extend its functionality and do it in a way that directly map onto GMail and the other offerings (not gonna happen).

What does the object store look like?

What is the object store behind the reader?

1. Is it a file system with a directory per feed, a file for the meta-information on that feed, a file for the feed entries, and a file for the search index? I know of one reader that does it that way.

2. Is it a true dynamic object store with the meta information stuck to each rss entry where the feed name and information are just meta-tags? doubtful.

3. Is it a database back end which is shared by multiple reader users and broken into complex table relations? Most likely, but it doesn’t have to be.

4. In the end does it really even matter? No it doesn’t.

Herein lies the biggest problem with most attempts at object store systems. They are attempts to capture the visualization and map it directly to the storage. The memory layout, visualization, storage, indexing, and security are all munged together into one tangled mess.

Design for storage, not for representation

The object store layout should not be a 1:1 mapping to a visual display. That is what we have with current file browsers, directories, and files. The file browser represents the file system implementation in all it’s grueling detail. No wonders you can’t find your files. On most linux systems file names which differ only in case are different names. Some love it, and some hate it. Does it really matter if there is a good visualization abstraction? No, the abstraction hides the implementation details that the user should not care about. there also needs to be an abstraction layer for the interface to the storage. Something for the developers to use which is simple and does not expose the implementation details of said storage.

Most people who start to tackle implementing an object store replacement for a file system take the reverse approach and attempt to make the object store implementation represent the particular visualization or visualizations they are thinking of. They design  the interfaces for storing the data to expose the details of the implementation. Exposing the meta-data as a core component. You need to first save to the temp storage, then push to the long term storage (for instance). They expect the storage to glean some important information from the meta-data (image dimensions, sampling rate, author, history). At some level this is needed (re: security), but most systems I have seen take this too far. What you want to do instead is figure out what the real issues are, and design the storage system for storage management; not for the meta-systems. The meta systems should be used by higher level abstractions for viewing that storage in interesting ways. This is extremely hard to do properly. When done properly, it does not matter if the data and meta-data is coming from a database, a file system, or a cloud. that becomes an implementation detail. Some systems will work better for different types of data and meta-data. That is how it should be. There are no silver bullets.

At some point I would love to go over how I think all this can be achieved. Where the levels of abstraction are, and what the meta-management systems would look like. The concepts are actually quite simple. Security First, Storage, Meta-Meta, Meta, Index, Visualization. Configuration management systems are the closest to a complete implementation I have seen. I just do not have time for all these ‘pet projects’ which are years of work….

It appears that the AZN cable channel will be going off the air. This is a little sad for me, as it was one of the few places I could find some of the movies friends have been recommending I see, with their original subtitles. The Anime they have is mixed at best (currently only Last Exile and Moon Phase), and the movies have been spotty. Guess its back to Netflix.

I must admit that I have spent a number of late evenings dancing with Sarah to the opening and ending music of Moon Phase and Last Exile. She prefers the former, I prefer the latter. I am a little ashamed to admit that I have seen all of Moon Phase in it’s subtitled form (way back when Josh was born no less so the full circle is kind of nice.) The dubbing is some of the worst I have heard in a long time (and I have heard some bad stuff.) The first time I saw Josh or Sarah smile were to that anime, so it grudgingly holds a place in my heart if for no other reason; well that and the neko mimi. Last Exile is decent enough with a great soundtrack, great writing, and above average animation.

The real loss are some of the movies that now I will just have to rent and plan for. I personally would rather stumble upon a movie that was recommended, but that has always been hit or miss anyway; more miss than hit really. I did get to spend some late nights working on PyCon-Tech, (or in one case, not working on it,) watching the channel. I tend to enjoy the surreal. Actually I revel in it. There is no better escape than a surreal movie or an on-stage farce. Soon after Sarah was born, the AZN channel aired Wool 100%, which came highly recommended from friends who share a similar affliction to mine. I spent the entire movie standing rocking Sarah. I was mesmerized from beginning to end. I missed the chance to see Kamikaze Girls again when they aired it, but it is another of those borderline surreal movies which scratch that itch; and I hate the entire goth-loli things; really, really hate it, all evidence in this post to the contrary. (Why Nana why!??! Your 2002 demo had so much promise!)

One of these days I will get around to making a list of good anime/manga. One of these days. Until then, go see Paprika! (seriously, go see it.)

Well it has been a very long day, so lets begin.

I woke up at 4:30am (a time I am more know for going to bed at), and got ready to be picked up by the airport limo. Because I planned for every eventuality, there turned out to be no need for such planning. There was no traffic. I got my bags checked immediately, and proceeded to the book store where I picked up Terry Pratchets latest book ‘Making Money’.  I have a trdition of reading Prattchet every time I fly. I will continue this tradition even after he stops witting, which may have unfortunately already be the case. But I digress. The flight was a short hop and I managed to sleep for about an hour of it (thank you DJ Tiesto - In Search of Sunrise 7). Landed, got my bag, and hopped on the free shuttle. Got to the hotel, and at 10:20am I was standing in my hotel room!

Then I made the mistake of trying to get the wireless working. Two calls to customer support, a wireless bridge, and secondary wireless card later, I was successfully connected to the wireless app, and crashing their servers (I have a habit of doing that).  And I am paying $13 a day for this privilege (more on this later). I am up and running, so I start dealing with the PyCon registrations. Nothing spectacular. There are some more vendor and press passes to be made up that didn’t make the press run, but no big deal over all. I connect up with David Goodger, Steve Holden, and Peter Kropf in the atrium and have a decent enough late lunch. To me its 3pm and I am starved.

Things seem to be going way too smoothly. The base network is in place, the power drops are in, and the pallets of printed materials are on the dock. I figure I will have time to get a hair cut after all. I have about 4 hours before I need to be back, so I promptly go up to my room and fall asleep for those 4 hours. Then some more reg work (up to 1005 pre-paid in full registrants!), and a peek at the schedule app. 680 people using the application!!! DAMN! Brett Cannons talk is at the top of the attendance list. By the app numbers we expect over 350 people to attend that talk. That is more attendance than all of Python 2004. That wakes me up enough to head down for what I expect to be an all night bag stuffing event. An event I have been greatly looking forward to.

It’s done. The bags (all 1000+) were stuffed in less than an hour. this is insane. Well it turns out that NONE of the sponsor materials are arriving at the hotel until tomorrow after noon. Yikes. So what as stuffed? 5 items. A far cry from the 21 we had last year, and the 8 shirts. Well we are not stuffing shirts this year. The solution is that we will try to have a separate packet to go with the shirts and hand those out Thursday evening and Friday. We might end up re-stuffing or something. Not really sure what the end decision is, and it will greatly depend on the amount of materials we have from the sponsors.

Things are a blur after that. People decided to merge the tutorial and non-tutorial badges into one poorly alphabetized pile. They did not know that the badges were broken into two sets on purpose, not that they were pre-sorted in a special way (due to being printed in sheets). Steve took this in stride and with some help we re-re sorted everything in short order.  The network is not 100% functional yet, but that will be fixed at 5:30 when the network guys get in and complete the job. We have a great group of people running the networking and Jaffo seems to like their work; thus I am quite impressed and have 0 concerns. The hotel room wireless is a different issue, and they are playing some dirty pool, which I will talk about another time.

There were some other small issues of little note. Meeting up with people I haven’t seen in a year or only know online. It will be good to see James Tauber again tomorrow, and I am looking forward to the tutorials. One issue is the registration desk which I will be running the on-site portion of at lest during the crunch.  We still don’t have a printer/computer pair which work together properly (my laptop does not like the generated PDF’s for some unknown reason), and there are ~40 unpaid registrations which we will have to deal with. The job of getting ~440 people through registration before 9am! That is going to be fun. I still have not seen Jeff or Mary to talk logistics, so we will see what happens 4 hours from now.

With that I am off for another 4 hour nap before registration setup starts!!!

Like most other children, our son has a favorite item, a little stuffed monkey, aptly named, monkey. Monkey went with us to Montana, and here is a Tabblo immortalizing the trip from his unique perspective.

See my Tabblo>

NOTE: There is a picture of us standing around a broken frostfree hidden in there.

Well I am sitting here in my hotel on the last evening I have internet access, eating Pocky, typing away as fast as I can, wondering where the week went. Tomorrow I move to a coffee shop before heading back to the ranch. In the week before the trip I was too overwhelmed with last minute stuff at work (like migrating to Python2.5), and preparing to spending 10 days without all my electronics. As a result I only posted information on the potter predictions site. There were a million little things which I figured I would get done while having 10 days with little to do but some minor farm work. This also meant that all the things I like to post about had major announcements. It all backed up, but I still figured 10 days away from the daily regiment would allow me to clear out some of the backlog. No such luck.

Read the rest of this entry »