In this brief teshuva, Rav Moshe rules that it is forbidden to listen to non-Jewish religious music, even on the radio or a recording. This applies not only to current songs, but to ones from the past as well [ed. note - this can be an issue when taking music courses in college and Gregorian chants are part of the curriculum]. Even if the song uses psukim from Tehillim (or elsewhere in Tanach, e.g. The Hallelujah Chorus), it is forbidden. However, if the song is written and/or song by a non-Jew but has no religious purpose, then there is technically no prohibition.
Rav Moshe considers the case of Acheir, who the Gemara claims left Jewish religious life as a result of his constantly singing Greek songs. After some discussion as to whether this could really be the reason that he abandoned his religious life, Rav Moshe cites the Maharsha who says that he was singing songs that had connections to idolatry that led him to heresy. Rav Moshe also notes that a potential issue is the mentioning of the names of other gods, as that would violate the prohibition of ושם אלהים אחרים...לא ישמע על פיך.
[ed. - Does that last point present a problem from songs such as "Let it Be" or "Walking in Memphis"? Discuss.]