Section 12. And be it further enacted by the authority aforesaid, That it shall be the duty of the Attorney-General immediately to remove from office any one of his deputies who shall refuse or neglect to perform the duties enjoined on him by this act.