I've documented the framework of this here: https://fedoraproject.org/wiki/User_Driven_Testing
As Colin Walters pointed out "This needs to go through the Fedora Design group.", and as Greg DeKoenigsberg wrote "I think it's big enough that it would be worth organizing a FAD for. It's exactly the kind of 'infrastructure of participation' solution that requires significant investment, but drives major change."
I agree whole-heartedly with both of them. So, let's do a FAD and see if over the course of a few days (possibly as much as 5), we can hash out a solid design and the basis of an implementation.
Schedule-wise, I'm looking at sometime between June and September 2010. If you're interested in participating, please either reply here, or let me know via email. We'll figure out where the best location for this is when we have an idea of who wants to participate. Also, if you want to participate, but have specific dates which will not work, be sure to state that too.
This isn't just for Red Hat folks either! If you want to help, please, chime in!