The New York Public Library Labs has built a new Rails-based usability testing tool called Infomaki. It' still a "zero-point” release, but an open source release is planned soon.
Read more about it at "Introducing Infomaki: Bite-sized Usability Testing."