Police are searching for survivors after a boat capsized off the coast of Malaysia, killing at least 13 people, officials say.