In an intriguing article, John Sietze Bergsma and Scott Walker Hahn argue that Ham’s sin against Noah in Genesis 9:20-27 was neither voyeurism, nor castration, nor paternal incest (solutions which have been proposed by various scholars), but instead was the sin of maternal incest.  An interesting treatment in its use of OT backgrounds, OT intertextuality, and literary criticism.  And, I think, a viable solution to a difficult passage.  Bergsma and Hahn provide this concluding summary: 

In the review of the various interpretive options for Gen 9:20–27 above, it has been seen that the voyeurist position, which understands Ham’s deed as nothing more than looking, fails to explain the gravity of Ham’s sin or the cursing of Canaan. The castration view suffers from a lack of textual support. The currently popular paternal-incest interpretation has much to commend it, but in almost every case the evidence marshaled for this view actually better suits the maternal-incest theory. The heuristic strengths of the maternal-incest interpretation are manifold: it explains (1) the gravity of Ham’s sin, (2) the rationale for the cursing of Canaan rather than Ham, (3) Ham’s motivation for committing his offense, (4) the repetition of “Ham, the father of Canaan,” and (5) the sexually charged language of the passage. In addition, biblical and ancient Near Eastern analogues for Ham’s crime are easy to find, and the related passages of the Pentateuch fit together more elegantly on this interpretation. 

 John Sietze Bergsma and Scott Walker Hahn, “Noah’s Nakedness and the Curse on Canaan (Genesis 9:20-27),” Journal of Biblical Literature 124/1 (2005): 39-40.  

You can read the entire article here.