32 to 64 bit remotely

Nathan Kinkade, July 15th, 2008

A couple months ago I posted here about some of our experiences with Varnish Cache as an HTTP accelerator. By and large I have been very impressed with Varnish. We even found that it had the unexpected benefit of acting as a buffer in front of Apache, preventing Apache from getting overwhelmed with too many slow requests. Apache would get wedged once it had reached it’s MaxClients limit, whereas Varnish seems to happily queue up thousands of requests even if the backend (Apache) is going slowly.

However, after a while we started running into other problems with Varnish, and I found the probable answer in a bug report at the Varnish site. It turns out that Varnish was written with a 64 bit system in mind. That isn’t to say that it won’t work nicely on a 32 bit system, just that you better not expect high server load, or else you’ll start running into resource limitations in a hurry. This left us with about 2 options: Move to 64 bit or ditch Varnish for something like Squid. Seeing as how I was loathe to do the latter, we decided to go 64 bit, which in any case is another logical step into the 21st century.

The problem was that our servers are co-located in data centers around the country. We didn’t want to hassle with reprovisioning all of the them. Asheesh did the the first remote conversion based on some outdated document he found on remotely converting from Red Hat Linux to Debian. It went well and we haven’t had a single problem on that converted machine since. Varnish loves 64bit.

I have now converted two more machines, and this last time I documented the steps I took. I post them here for future reference and with the hope that it may help someone else. Note that these steps are somewhat specific to Debian Linux, but the concepts should be generally applicable to any UNIX-like system. There are no real instructions below, so you just have to infer the method from the steps. See the aforementioned article for more verbose, though dated, explanations. BE WARNED that if you make a mistake and don’t have some lovely rescue method then you may be forced to call your hosting company to salvage the wreckage:

• [ssh server]
• aptitude install linux-image-amd64
• reboot
• [ssh server]
• sudo su -
• aptitude install debootstrap # if not already installed
• swapoff -a
• sfdisk -l /dev/sda # to determine swap partition, /dev/sda5 in this case
• mke2fs -j /dev/sda5
• mount /dev/sda5 /mnt
• cfdisk /dev/sda # set /dev/sda5 to type 83 (Linux)
• debootstrap –arch amd64 etch /mnt http://http.us.debian.org/debian
• mv /mnt/etc /mnt/etc.LOL
• cp -a /etc /mnt/
• mv /mnt/boot /mnt/boot.LOL
• cp -a /boot /mnt/ # this is really just so that the dpkg post-install hooks don’t issue lots of warnings about things not being in /boot that it expects.
• chroot /mnt
• aptitude update
• aptitude dist-upgrade
• aptitude install locales
• dpkg-reconfigure locales # optional (I selected All locales, default UTF-8)
• aptitude install ssh sudo grub vim # and any other things you want
• aptitude install linux-image-amd64
• vi /etc/fstab # change /dev/sda5 to mount on / and comment out old swap entry
• mkdir /home/nkinkade # just so I have a home, not necessary really
• exit # get out of chroot
• vi /boot/grub/menu.lst # change root= of default option from sda6 to sda5
• reboot
• [ssh server]
• sudo su -
• mount /dev/sda6 /mnt
• chroot mnt
• dpkg –get-selections > ia32_dpkg_selections
• exit
• mv /home /home.LOL
• cp -a /mnt/home /
• mv /root /root.LOL
• cp -a /mnt/root /
• mkdir /mnt/ia32
• mv /mnt/* /mnt/ia32
• mv /mnt/.* /mnt/ia32
• cp -a bin boot dev etc etc.LOL home initrd initrd.img lib lib64 media opt root sbin srv tmp usr var vmlinuz /mnt
• mkdir /mnt/proc /mnt/sys
• vi /mnt/etc/fstab # make /dev/sda6 be mounted on / again, leave swap commented out
• vi /boot/grub/menu.lst # change the default boot option back to root=/dev/sda6
• reboot
• [ssh server]
• sudo su -
• mkswap /dev/sda5
• vi /etc/fstab (uncomment swap line)
• swapon -a
• dpkg –set-selections < /ia32/ia32_dpkg_selections
• apt-get dselect-upgrade # step through all the questions about changed /etc/files, etc.

SemanticEditTime

Steren Giannini, July 14th, 2008

My last task concerning Semantic Media Wiki was to create an extension which adds automatically the “Revision Time” and the “Revision User” of a wiki page in the semantic data.

You can now use these data to sort querries, for example :

{{#ask: [[Category:Task]]
| sort=Revision time
| order=descending
| limit=3
}}

It’s currently used in the Casestudies page of the CC wiki.

You’ll find the code and the instructions in the Creative Commons Git repository. This small piece of code is more a hack than a real extension. The semantic properties are supposed to be rewritten in the next version of Semantic MediaWiki, which will enable much easier user-defined special properties.

Flickr Image Re-Use for OpenOffice.org Demo availlable

Mihai Husleag, July 12th, 2008

Never trust a programmer when he gives you a date for something to be done . Thats what i did in my last article (2 weeks i think i said then) and here we are a month later.

What has be done since my last article :

• right click on an result (image) will show you a popup menu with the available sizes on Flickr server
• left click on a result will insert directly in Writer the image with size medium as default
• once the image is inserted some text will be added beside the image(title , link to the image, license and link to the license)
• i improved searching and the way the image is adding into Writer
• i added a more friendly interface when you want to search over license (similar with advanced search on Flickr website)
• added a previous button to see previous results if needed
• if you insert a image, when you open again the extension the previous search will be done immediately ( on the exact the same position if you used the previous or next buttons)
• about searching : we can have multiple tags (separated by space : ” “) and the relation between them is AND . also the results are ordered by interestingness
• a progress bar was added
• the extension after installation can be found here : Insert \ Picture \ From Flickr …

Some screenshots :

The results from a search by the extension vs Flickr search

Right click on an image

Inserted image into Writer

Also i would to add that this extension, at this momment, works only in Writer.

Download (right click and save as)

Any suggestions or remarks are greatly appreciated.

License-oriented metadata validator and viewer: libvalidator

Hugo Dworak, July 8th, 2008

As the Google Summer of Code 2008 midterm evaluation deadline is approaching, it is a good time to report the progress when it comes to the license-oriented metadata validator and viewer.

The source code is located in two dedicated git repositories. The first being validator, which contains the source code of the Web application based on Pylons and Genshi. The second repository is libvalidator, which hosts the files that constitute the core library that the project will utilise. This is the component that the development focuses on right now.

The purpose of the aforementioned library is to parse input files, scan them for relevant license information, and output the results in a machine-readable fashion. More precisely, its workflow is the following: parse the file and associated RDF information so that a complete set of RDF data is available, filter the results with regard to license information (not only related to the document itself, but also to other objects described within it), and return the results in a manner preferable for the usage by the Web application.

pyRdfa seems to be the best tool for the parsing stage so far. It handles the current recommendation for embedding license metadata (namely RDFa) as well as other non-deprecated methods: linking to an external or embedded (using the “data” URL scheme) RDF files and utilising the Dublin Core. The significant lacking is handling of the invalid direct embedding of RDF/XML within the HTML/XHTML source code (as an element or in a comment) and this is resolved by first capturing all such instances using a regular expression and then parsing the data just as external RDF/XML files.

Once the RDF triples are extracted, one can use SPARQL to narrow the results just to the triples related to the licensed objects. Both librdf and rdflib support this language. Moreover, the RDF/XML related to the license must be parsed, so that its conditions (permissions, requirements, and restrictions) are then presented to the user.

The library takes advantage of standard Python tools such as Buildout and nose. When it is completed, the project will be all about writing a Web application that will serve as an interface to libvalidator.

Juggling Metadata at CC

Nathan Yergler, July 3rd, 2008

It’s the day before a long weekend and inspiration hits — RDFa and ccREL are exactly like juggling!

Also available: MP4, Ogg.

In other news of the weird, I was actually able to use PiTiVi to trim the video up after shooting it on my phone. Cool.

RDFa for Semantic MediaWiki [GSoC 2008]

David McCabe, July 1st, 2008

Hello, world!

My name is David McCabe, and this summer I am adding RDFa support to Semantic MediaWiki, as part of the Google Summer of Code 2008. I am an undergraduate in Mathematics at Portland State University. For the Google Summer of Code 2006, I wrote Liquid Threads, a MediaWiki extension that replaces talk pages with a threaded discussion system.

Semantic MediaWiki (SMW) is the software used for the CC wiki and many other wikis. SMW allows authors to mark up wiki pages so that their contents and relationships are machine-readable. SMW already publishes this machine-readable data in RDF/XML format.

You can read about RDFA on the CC Wiki. There is also a Google Tech Talk on RDFa.

Per-blog registrations for Wordpress-mu

Nathan Kinkade, July 1st, 2008

About a year ago when I started at Creative Commons, I was asked to write a Wordpress-mu plugin that would allow users to register with specific sub-blogs and not just the root/default blog. So I wrote a plugin that allowed that to happen and also fixed a few inconsistencies that I noticed in the code at that time.

Recently I reviewed my old posts at the Wordpress-mu forums and realized that there was some activity on those posts that I hadn’t noticed. It became clear that the plugin needed some modification and also that it only supported per-blog registrations for Wordpress-mu installs based on subdomains, but didn’t work for installs based on subdirectories.

This past weekend I took another look at the plugin and found that most of the inconsistencies I had been seeing, and for which I had tried to code around in the plugin, were gone in v1.5.1, and so I was able to remove the vast majority of the plugin’s code.

The current plugin seems to work for both subdomain-based and subdirectory-based installs, at least on a fresh install of Wordpress-mu that I was working with.

Update [Mon Aug 17 14:02:16 EDT 2009]: http://code.creativecommons.org/issues/issue436.

Labs and Techblog have merged

Nathan Kinkade, June 28th, 2008

For most people reading this, you will have likely already realized that the CC Techblog has moved to Labs. We recently realized that Labs and the Techblog overlapped enough in intent to merge them into one mostly cohesive interface. The name Techblog has gone away in favor of the name Labs. The front page of Labs is now what was the Techblog. The new Labs blog is also sporting a new old theme, which is the same theme used at http://creativecommons.org with a few minor changes.

No more excuses for not doing tasks!

Steren Giannini, June 24th, 2008

Stay up to date and don’t miss your deadline any more with the brand new email task notifications and reminders.

I created this as an extension of Semantic MediaWiki. It is mainly supposed to be used as an internal tool at CC. Basicaly, it sends email to concerned users each time a tasks is created or updated. It also sends reminders at different date before your deadline.

I’m currently working on an online release of the entire project and task tracking system used here, so stay tuned.

One of those days…

Frank Tobia, June 23rd, 2008

I wake up in the morning to find that eternal funny man George Carlin has died. Add to that the fact that one of my recently-ported api tests has failed, and you just know it’s going to be one of those days.

Work on porting the test suite was slow at first. I had thought it would be easy enough just to jump in and start coding. Later this pipe dream evaporated and I forced myself to learn CherryPy, Python Paste, and a bit of WSGI.  Then Asheesh and Nathan filled me in on the intricacies of the “buildout” build system (which is really quite nice when you get to know it), and I was ready to go.

So the porting has begun. Check out the branch where all the fun is happening, if you happen to be so inclined. And as the fates would have it, the lucky number seventh test I ported just so happens to fail. This leads to the real challenge every software tester must eventually face: fixing a broken test. Getting to the bottom of this means a fun-filled day ahead.