We don't know any good Rock Band 3 open-source alternatives yet.
You are more than welcome to suggest any alternatives that you know of, or ask the community for suggestions.
Alternatively, we recommend checking out LibHunt for open-source libraries and alternatives to Rock Band 3.