Oil hole cap with cover for oil hole cap