Sit down and talk with him.
If he really has low sex drive in general then it is okie. But if
he has low sex drive with you only but can lust after other girls
(kept thinking about having sex with them), then there might be a
deeper issue that you need to address in the rs.
