This week it was Scotty’s turn to be on the road, so in the absence of our DA (Designated Adult), the RDC crew did some experimenting with Steven’s new auto-downloader program and looked at how to create a collection for the XO. We ran a few tests downloading some of the subject groups in our target collection – and the program is working splendidly! We also spent some time looking at the new S3 API recently released by the Internet Archive as yet another alternative means of downloading the IACL collection (in addition to the Open Library Bulk Access Downloader). We plan to use our auto-downloader – but it is always good to have a backup plan.

Ideally, we want to make the creation of these bundles as automated as possible, and extend the ability of the downloader to retrieve materials from any source so anyone who wants to can start creating bundles for the OLPC XO. Right now, Steven is exploring to see if he can get his auto-downloader to create the bundles according to the structure format outlined on the OLPC wiki, auto-generating the necessary directories and library files. He is also developing a roadmap of pre- and post-launch deliverables for his app so he can continue development beyond the RDC Summer Mentoring program. We plan to host his code and continue to help him with his quest! Levi is pitching in with the first order of design business: creating a custom icon that will represent our IACL-4-OLPC bundle like the ones at wiki.laptop.org.

We also brainstormed the idea of downloading the books in bulk (IACL-complete) as well as in the organized sub-collections (IACL-by-topic) as represented in our target collection. As we mentioned previously, our target collection is essentially a subset of the IACL. This target collection was also used in another open source project called the Virtual Shelf, created by two talented developers from the Berkeley School of Information (their prototype application only runs in Mozilla Firefox 3. If you cannot view it, you can read more about it in the developer’s master thesis). We want to retain the meta-information if possible (at this point, just the sub-collection information from the Internet Archive), but also give others the ability to build their collection from scratch. The idea is to pass this collection on to others – not just the OLPC, but to other folks with XOs – and they should be responsible for their own librarianship. We do want to do as much as possible to help get them started.

Speaking of passing on the collection … we will be picking up some of the area XOs this week. First stop: our friends at the Port Orford Library!

Greetings from Rebecca Malamud on behalf of the Rural Design Collective! Apologies for this delayed #rdc2009 Hacker Wednesday report – I am on the road again visiting friends and family – but always in touch with the RDC crew in Oregon via Skype and email during the trip. We are starting to make real progress on the IACL-4-OLPC project!

Steven has successfully downloaded his first book with his new auto-downloader program! Written fully in python, it can retrieve a user-defined list of books and specify what directory the files are being saved to. Scotty Auble, our Development Mentor for the IACL-4-OLPC project, states that this differs from the Bulk Access Downloader available from Open Library in that it uses straight python urlib calls instead of relying on linux/unix wget commands. We are investigating if there is a way to still use rate limiting to decrease the load to the IA servers when downloading the books. Since we are only dealing with a subset of the Internet Archive Children’s Library (2,000 out of 3,322 books), and the examples set forth on the Open Library blog are 700K+, we are inclined to believe that impact on the network will be marginal. But we plan to check with folks at the Archive just because it is good manners .

Steven, who has code-named his application “Oghams Prayer”, has set some new personal goals on the project – he wants to (1) auto-detect file formats in the list so it does not have to be hard-coded, (2) develop a means to sort and file books based on topic (we built our collection based on the IACL Tag Cloud so we would like to retain that meta-information, another reason to not just do a wget), and (3) make the application extensible so it can download books from multiple sources, not just the IACL. He also plans to develop a simple user interface, and release the code as open source. The RDC team will be helping him with the rollout of his new application!

Levi has wrapped up his first draft of our DjVu Memory Chart – and mastered some new CSS and HTML skills in the process. Ever the skeptics, we were curious to see how much the DjVu memory per page varied on different types of books. It is obvious that this can vary a great deal depending on the content, so it will be interesting to see how much memory the IACL books we have targetted will actually occupy. We are estimating that an 8GB memory stick will be more than enough. We do plan on finessing our sample collection a bit, both in terms of design and content, so it might provide a useful guide for someone else planning to build a collection in the DjVu format.

It was a busy and successful week! Next up: downloading our IACL collection and exploring the EPUB format! We also have a special surprise announcement that will be revealed at a later date. Stay tuned to the RDC!