It often happens that, when you attempt to remove a COM Add-In from an Office application, it shows a message saying that 'This add-in is installed for all users on this computer and can only be connected or disconnected by an administrator.'.
If you face this case, it is because the said add-in was installed using admin permission. To disconnect that add-in, you need to follow the steps as mentioned in this post.
When you install an add-in, it asks to install either for all the users of that system or for the current user. The default value of the selection is set by the developer who generated the installation package.
In case you installed it for all users, which generally needs admin permission, then you need admin rights while disabling the said add-in. Otherwise, the following message will prompt on the screen and disallow you from proceeding further:
To resolve this issue, close Outlook first. Then right-click on the Outlook icon and click 'Run as administrator' from the context menu. Alternatively, you can press CTRL + ALT + SHIFT and click on the Outlook icon. UAC (User Access Control) popup will be shown, asking you to allow administrative rights. It may ask you to enter admin credentials, if the currently loggedd-in user is a standard user.
After the above steps, try removing/disconnecting the add-in again. You will now be allowed to perform the same.