Olive oil is an excellent makeup remover. Just not for acne prone skin.
Olive oil is moderately comedogenic, meaning it is somewhat likely to cause pimples. If your skin breaks out with most other products, then olive oil will almost surely cause breakouts. You may not see sudden pimples, but say the next morning or in a week, if you use it daily.
Thorough cleansing to remove all traces of it after removing makeup does help but it is not fool proof. Because of this we cannot confidently recommend olive oil as a makeup remover for acne prone skin.
If you do want to give it a shot, look for the thinnest consistency (most liquidy) that you can find, not extra virgin, not pomace, just the regular light green variety.
Mineral oil available as Johnson’s Baby Oil is a non-comedogenic oil that can be used to remove even water proof makeup. Still, cleansing with a face wash is necessary afterwards.
If you want to use only oils of vegetable origin (mineral oil is not), try Aroma Magic’s Jojoba Oil. It is easily available although not cheap. Jojoba oil has the reputation for being safe, even good for acne prone skin. Do a patch test before using it all over.
Steer clear of castor oil.