We don't know any good Hexen: Beyond Heretic open-source alternatives yet.
You are more than welcome to suggest any alternatives that you know of, or ask the community for suggestions.
Alternatively, we recommend checking out LibHunt for open-source libraries and alternatives to Hexen: Beyond Heretic.