Follow the answer: http://sharepoint.stackexchange.com/questions/156521/error-the-referenced-file-catalogs-mycustompage-master-is-not-allowed-on-th
It seems the masterpage needs to be included in the project.
The masterpages under /_catalogs/masterpage are publishing masterpages, they are stored in the Content DB. It seems for application pages the server only searches for masterpages under the _layouts folder.