We had an incredibly productive week at RDC headquarters this week, working remotely and on-site. James Simmons, author of the View Slides 8 and Get Internet Archive Books 2 (GIAB) Activities for Sugar contributed his considerable knowledge and expertise in this area. Via email, we brainstormed the best approach for making these books accessible – what we are trying to do is unique, in that we are trying to provide a completely offline solution so that children without an Internet connection can enjoy these books. The GIAB program that Jim has written is the online equivalent of what we are trying to do. Ideally, we would like to author our own Activity, but it just makes sense to build on a good code base and tap into a knowledgeable resource. That is how the Internet and open source works.

We have learned that it is not feasible to create a content bundle with too many items because it simply overloads the memory of the XO when it attempts to load all of the contents of the index into the Journal. As a matter of fact, 5 – 20 MB is the recommended bundle size, which is ridiculous when you are dealing with the DjVu or PDF format (Epub, of course, is another kettle of fish – but we will get to that later). There could only be 1-2 books per bundle which (needless to say) would make browsing the collection quite difficult. So, we are looking into methods to either extract a single book from a bundle or store them on the stick as individual files using GIAB as the interface to navigate the collection. Jim even suggested that what we are creating aren’t technically bundles and should perhaps have a different name altogether – such as “Children’s Library Archive”. Jim created a branch for us on github and we will be taking a close look at his code to see what is possible.

We are also making great progress on the web user interface and that will begin to come together in the next week or so. Levi has been developing mockups for browsing the collections, and creating an icon set reflecting the different categories. We want to make it as easy as possible for a user to view the book online or download and install the book onto their XO. We are also looking at AJAX libraries and the best tool to collect the metadata – Open Library API versus Internet Archive Advanced Search.

We view all of this as a work in progress – and have created plenty of work to challenge us in mentoring programs to come!

This week it was Scotty’s turn to be on the road, so in the absence of our DA (Designated Adult), the RDC crew did some experimenting with Steven’s new auto-downloader program and looked at how to create a collection for the XO. We ran a few tests downloading some of the subject groups in our target collection – and the program is working splendidly! We also spent some time looking at the new S3 API recently released by the Internet Archive as yet another alternative means of downloading the IACL collection (in addition to the Open Library Bulk Access Downloader). We plan to use our auto-downloader – but it is always good to have a backup plan.

Ideally, we want to make the creation of these bundles as automated as possible, and extend the ability of the downloader to retrieve materials from any source so anyone who wants to can start creating bundles for the OLPC XO. Right now, Steven is exploring to see if he can get his auto-downloader to create the bundles according to the structure format outlined on the OLPC wiki, auto-generating the necessary directories and library files. He is also developing a roadmap of pre- and post-launch deliverables for his app so he can continue development beyond the RDC Summer Mentoring program. We plan to host his code and continue to help him with his quest! Levi is pitching in with the first order of design business: creating a custom icon that will represent our IACL-4-OLPC bundle like the ones at wiki.laptop.org.

We also brainstormed the idea of downloading the books in bulk (IACL-complete) as well as in the organized sub-collections (IACL-by-topic) as represented in our target collection. As we mentioned previously, our target collection is essentially a subset of the IACL. This target collection was also used in another open source project called the Virtual Shelf, created by two talented developers from the Berkeley School of Information (their prototype application only runs in Mozilla Firefox 3. If you cannot view it, you can read more about it in the developer’s master thesis). We want to retain the meta-information if possible (at this point, just the sub-collection information from the Internet Archive), but also give others the ability to build their collection from scratch. The idea is to pass this collection on to others – not just the OLPC, but to other folks with XOs – and they should be responsible for their own librarianship. We do want to do as much as possible to help get them started.

Speaking of passing on the collection … we will be picking up some of the area XOs this week. First stop: our friends at the Port Orford Library!

Greetings from Rebecca Malamud on behalf of the Rural Design Collective! Apologies for this delayed #rdc2009 Hacker Wednesday report – I am on the road again visiting friends and family – but always in touch with the RDC crew in Oregon via Skype and email during the trip. We are starting to make real progress on the IACL-4-OLPC project!

Steven has successfully downloaded his first book with his new auto-downloader program! Written fully in python, it can retrieve a user-defined list of books and specify what directory the files are being saved to. Scotty Auble, our Development Mentor for the IACL-4-OLPC project, states that this differs from the Bulk Access Downloader available from Open Library in that it uses straight python urlib calls instead of relying on linux/unix wget commands. We are investigating if there is a way to still use rate limiting to decrease the load to the IA servers when downloading the books. Since we are only dealing with a subset of the Internet Archive Children’s Library (2,000 out of 3,322 books), and the examples set forth on the Open Library blog are 700K+, we are inclined to believe that impact on the network will be marginal. But we plan to check with folks at the Archive just because it is good manners .

Steven, who has code-named his application “Oghams Prayer”, has set some new personal goals on the project – he wants to (1) auto-detect file formats in the list so it does not have to be hard-coded, (2) develop a means to sort and file books based on topic (we built our collection based on the IACL Tag Cloud so we would like to retain that meta-information, another reason to not just do a wget), and (3) make the application extensible so it can download books from multiple sources, not just the IACL. He also plans to develop a simple user interface, and release the code as open source. The RDC team will be helping him with the rollout of his new application!

Levi has wrapped up his first draft of our DjVu Memory Chart – and mastered some new CSS and HTML skills in the process. Ever the skeptics, we were curious to see how much the DjVu memory per page varied on different types of books. It is obvious that this can vary a great deal depending on the content, so it will be interesting to see how much memory the IACL books we have targetted will actually occupy. We are estimating that an 8GB memory stick will be more than enough. We do plan on finessing our sample collection a bit, both in terms of design and content, so it might provide a useful guide for someone else planning to build a collection in the DjVu format.

It was a busy and successful week! Next up: downloading our IACL collection and exploring the EPUB format! We also have a special surprise announcement that will be revealed at a later date. Stay tuned to the RDC!