Instead of adding the classname oe_form_invisible to hide it, simply don't render the button if it shouldn't be displayed.
bf1e999