Caching deeds for peak performance

Nathan Yergler, January 6th, 2010

As Chris mentioned, he’s been working on improving the license chooser, among other things simplifying it and making it a better behaved WSGI citizen. That code also handles generating the license deeds. For performance reasons we like to serve those from static files; I put together some details about wsgi_cache, a piece of WSGI middleware I wrote this week to help with this, on my personal blog:

The idea behind wsgi_cache is that you create a disk cache for results, caching only the body of the response. We only cache the body for a simple reason—we want something else, something faster, like Apache or other web server, to serve the request when it’s a cache hit. We’ll use mod_rewrite to send the request to our WSGI application when the requested file doesn’t exist; otherwise it hits the on disk version. And cache “invalidation” becomes as simple as rm (and as fine grained as single resources).

You can read the full entry here, find wsgi_cache documentation on PyPI, and get the source code from our git repository.

Understanding the State of Sanity (via whiteboards and ascii art)

cwebber, December 18th, 2009

Since I started working at Creative Commons a number of months ago, I’ve been primarily focused on something we refer to as the “sanity overhaul”.  In this case, sanity refers to try and simplify what is kind of a long and complicated code history surrounding Creative Commons’ licenses, both as in terms of the internal tooling to modifying, deploying, and querying licenses and the public facing web interfaces for viewing and downloading them.  Efforts toward the sanity overhaul started before I began working here, executed by both Nathan Yergler and Frank Tobia, but for a long time they were in a kind of state of limbo as other technical efforts had to be dedicated to other important tasks.  The good news is that my efforts have been permitted to be (almost) entirely dedicated toward the sanity overhaul since I have started, and we are reaching a point where all of those pieces are falling into place and we are very close to launch.

To give an idea of the complexity of things as they were and how much that complexity has been reduced, it is useful to look at some diagrams.  When Nathan Kinkade first started working at Creative Commons (well before I did), Nathan Yergler took some time to draw on the whiteboard what the present infrastructure looked like:

as well as what he envisioned the “glorious future” (sanity) would look like:

When I started, the present infrastructure had shifted a little bit further still, but the vision of the “glorious future” (sanity) had mostly stayed the same.

This week (our “tech all-hands week”) I gave a presentation on the “State of Sanity”.  Preparing for that presentation I decided to make a new diagram.  Since I was already typing up notes for the presentation in Emacs, I thought I might try and make the most minimalist and clear ASCII art UML-like diagram that I could (my love of ASCII art is well known to anyone who hangs out regularly in #cc on Freenode).  I figured that I would later convert said diagram to a traditional image using Inkscape or Dia, but I was so pleased with the end result that I just ended up using the ASCII version:

*******************
* CORE COMPONENTS *
*******************

      .--.
     ( o_o)
     /'---\
     |USER| --.
     '----'   |
              |
              V
         ___   .---.
       .'   ','     '.
     -'               '.
    (     INTARWEBS     )
     '_.     ____    ._'
        '-_-'    '--'
              |
              |
              V
      +---------------+  Web interface user
      |   cc.engine   |  interacts with
      +---------------+
              |
              |
              V
      +---------------+  Abstraction layer for
      |  cc.license   |  license querying and
      +---------------+  pythonic license API
              |
              |
              V
      +---------------+  Actual rdf datastore and
      |  license.rdf  |  license RDF operation tools
      +---------------+  

****************
* OTHER PIECES *
****************

  +--------------+
  |  cc.i18npkg  |
  | .----------. |
  | | i18n.git | |
  +--------------+

********************************************
* COMPONENTS DEPRECATED BY SANITY OVERHAUL *
********************************************

  +------------+  +-----------+  +---------+  +-------------+
  |    old     |  | old zope  |  | licenze |  | license_xsl |
  | cc.license |  | cc.engine |  +---------+  +-------------+
  +------------+  +-----------+

This isn’t completely descriptive on its own, and I will be annotating as I include it in part of the Sphinx developer docs we are bundling with the new cc.engine.  But I think that even without annotation, it is clear how much cleaner the new infrastructure is at than the old “present infrastructure” whiteboard drawing, which means that we are making good progress!

One click PayPal donations with CiviCRM

Nathan Kinkade, November 9th, 2009

About a month ago CC launched its annual Fall fundraising campaign. Along with it we also rolled out a streamlined donation process. I wrote about this on the CiviCRM blog, and also wrote up some documentation on the CC Wiki. This new donation method required some custom code, and leveraging an existing CiviCRM script written by Donald Lobo.

CC @ Mozilla Service Week

Nathan Yergler, September 9th, 2009

Next week is Mozilla Service Week and Creative Commons is participating by hosting a week long help desk in IRC. You can find more details on our earlier blog post or in the wiki. Several CC staff members and community volunteers will be available during the week to answer questions about using CC licenses and the associated tools. We’ll be answering questions about:

• General CC help
• CC technology (ccREL and software projects)
• Where and how to publish CC works
• Where and how to find CC works
• CC in education and science

If you’d like to help out and educate others about using CC licenses and tools, you can sign up on the wiki page.

Creative Commons Drupal Module — GSoC 2009

Blaise Alleyne, September 3rd, 2009

This past year was my last at the University of Toronto, making this summer my last chance to participate in the Google Summer of Code. I searched hard for a project and mentor organization that would suit my interests, and when I noticed that the Creative Commons Drupal module was in need of some developer love, I knew exactly what I wanted to spend my summer doing. With John Doig as my CC mentor, and Kevin Reynen (the module’s maintainer and initial author) as an unofficial Drupal mentor, I’ve been privileged to have spent the past few months updating and extending the module.

A couple years ago, development for Drupal 4.7 was begun, but it was never quite completed. CC Lite came to be the reliable choice for Drupal 6. However, CC Lite’s scope is limited — it allows you to attach a license to content in Drupal, but that’s about it. The main CC module’s vision is broader — to fully integrate CC technology with the Drupal platform — and I hope I’ve helped to realize that just a little.

Some of the module’s features:

• it uses the CC API for license selection and information (so, for example, when new license versions are released, they become available on your Drupal site automatically)
• you can set a site-wide default license/jurisdictoin, and user’s can set their own default license/jurisdiction
• ccREL metadata is supported, output in RDFa (and, optionally, RDF/XML for legacy systems)
• supports CC0, along with the 6 standard licenses and the Public Domain Certification tool
• you can control which licenses and metadata fields are available to users
• basic support for the Views API has been added (including a default /creativecommons view)
• there’s a CC site search option

The module is still listed as a beta release, as some folks have been submitting bug fixes and patches over the past few weeks, though it’s quite usable. Special thanks to Turadg Aleahmad, who’s helped with a lot of the recent bug fixes towards the end of the GSoC term, and committed to being active in future development. If you’re into Drupal development, we could use help with testing, and any translations would be greatly appreciated too.

Right now, the focus is on getting to a stable release, but we’ve got lots of ideas for the future too. Thanks to John and Kevin for their support through the summer, and to Turadg for his recent help. I look forward to seeing the module put to good use!

Check it out!

I’m a musician, writer, software developer, free culture / free software advocate and recent graduate of the University of Toronto — get in touch at http://blaise.ca/

Developers landing page revamp

Greg Grossmeier, August 11th, 2009

If you haven’t looked at the Developers landing page on the Creative Commons wiki lately, you’re missing out! We’ve recently put a lot of effort into reorganizing the information, making the important things easier to find, and overall just making the whole place a bit more welcoming.

First of all, we’ve made the semantic split between information for desktop-based development and web-based development. At each page there is a list (and short description) of the various tools to help you integrate CC-license metadata functionality and a short list of open Developer Challenges. These challenges are things the developer community think would be cool to have; a wishlist everyone can help with!

Also, we’re starting to produce some more “HowTo” guides for developers who are interested in the best practices of integrating CC-license metadata. Thus far we have one for Web Integration which lists the various ways a service could support CC licenses with best practices examples (pictoral and code) of how they did it. See, for example, the page on adding license choice when uploading content.

We hope this redesign will make it easier for developers to find the information they need to improve their services. If you have any other suggestions, don’t hesitate to send an email to greg@creativecommons.org

License Engine path changes

Nathan Yergler, July 21st, 2009

We just pushed out a resolution to Issue 381, updating the path to the license engine. The license engine (the cc.engine application) is responsible for handling the license selection portions of the website as well as the deeds. Some background may be helpful.

When we launched CC0 earlier this year we made a conscious decision to locate the deed on a different path than the rest of the license deeds. CC0 is a waiver, not a license, so the deed appropriately is located at http://creativecommons.org/publicdomain/zero/1.0/ (as opposed to the license deeds which URLs that look like http://creativecommons.org/licenses/by/3.0/). At the time we decided to leave the CC0 chooser in the same group as the other choosers. This unfortunately gave us a URL that contained license in it — http://creativecommons.org/license/zero/ — which caused confusion for some users. Is it a license or isn’t it?

It’s not.

So starting this afternoon we’ve relocated the license engine to http://creativecommons.org/choose and CC0 to http://creativecommons.org/choose/zero. Redirects are in place so you shouldn’t notice any changes.

git-svn with svn:externals

Nathan Yergler, July 21st, 2009

We’ve been slowly but surely moving projects from Subversion to git, but there are still large pieces of code that are sort of deadlocked in Subversion. We make extensive use of svn:externals in our repository as a way to pull in dependencies and shared code (both within our repository and from other repositories, like zc.buildout’s bootstrap). This means that in order to convert something like the license engine (cc.engine) we need to also convert cc.license (which uses license_xsl), license.rdf and i18n. Of course, the API also uses license_xsl and the main site uses license.rdf. Taking the time to move all of that wholesale isn’t something we have the time or desire to do right now. It’s not just converting the repository — that’s the easy part; converting the deployments and surrounding tools is the real pain.

Last week I decided I wanted to use git to work on code currently in Subversion (my supporting tool chain really is that much better for git) and decided to check it out using git-svn. And once again I was burned by our use of svn:externals. So I wrote gsc — git subversion clone. You can read more details on my blog or find the code on gitorious.

Incidentally, cleaning up that dependency graph is very much on our radar. I’m hoping to land work that we started last summer this fall that will remove some duplicated code and clean up the dependencies of the remaining code.

OpenOffice.org Add-in Updates – GSoC 2009

NiMaL, July 6th, 2009

I’ve been working on the GSoC 2009 project, where I’m working on certain updates to the existing Creative Commons Add-in for OpenOffice.org. This project is mentored by Nathan Yergler.

The main goals for this project is to provide the following updates to the existing plugin.

• Update the codebase to the OOo 3 SDK
• The license selection UI could be refined to provide help around what each option means (”what is Share Alike”).
• Display license information when opening CC licensed documents
• Internationalization – prepare the code for translation and write the scripts to integrate PO files prepared by translators
• Support for OOo Draw
• Add support for CC0
• Make a release incorporating Flickr Image Re-Use for OpenOffice.org.

As there are many distinct tasks, I’m working on complete the most I can. I have been struggling a bit in the past couple of weeks with my progress, but I’m now bouncing back and coming back to track.

I hope I’ll be able to present the CC community with an updated version of the Add-in at the end of the project completion.

Updates to CC Network OpenID Provider

Nathan Yergler, April 13th, 2009

We received a couple bug reports over the weekend about the CC Network OpenID provider. Users were seeing a situation where they were asked to log into the CC Network with a username that was similarto their’s, but not quite right — specifically, the final character was omitted. When they put in their correct username and password they were just redirected to the login screen again and again.

After some digging we arrived at Issue 308. The problem occurs when a non-compliant OpenID-enabled site asks for the OpenID URL and the user supplies something that should work, but isn’t canonical. For example, my CC Network profile is at https://creativecommons.net/nathan/. I should be able to use any of the following as my OpenID and have the site get to the canonical version:

• creativecommons.net/nathan
• http://creativecommons.net/nathan
• http://creativecommons.net/nathan/
• https://creativecommons.net/nathan

The process of getting from one of these to the canonical version is called normalization.

So if an OpenID enabled site (such as, say, sourceforge.net or intensedebate.com) doesn’t do normalization, the user ends up asking to be validated for some URL other than their own canonical identity. And our server kicks that back as an invalid OpenID URL. Intense Debate seems to be aware of the issue; someone already reported it on their Get Satisfaction forum.

We briefly considered working around this bug (and it is a bug) but ultimately decided against it. It didn’t “smell” right and after some thought we realized that it could cause users more problems down the road. For example, if we implemented the proposed fix (making the trailing slash optional, which would have fixed it in most cases, it appears), a user could end up logging in at a broken site with the URL https://creativecommons.net/foo. If that site later fixed their code to correctly perform normalization, the user would suddenly be asking to validate a different URL — https://creativecommons.net/foo/. I suppose a site that has this bug and needs to fix it could normalize all the OpenID URLs stored in their database before pushing out the patch.

Instead we went ahead and added an error screen so that instead of the completely frustrating never-ending login loop, you at least get a hint that something’s wrong (and how to fix it).

The moral of the story? If you run into a problem logging into an OpenID-enabled site, first make sure you’re using your “real” URL (and if you still have a problem, at least with your CC Network ID, please report it).