Thanks HolyGuacamole. Looks like you answered both the question I asked and the one I didn't.
The one I asked: "yes, you have to re-work the alerts one-by-one"
The one I didn't: "can't prevent a "not up" component monitor from flagging the app monitor as "down" in the current release"
Thanks again, the 2nd question was actually even more important.