You wouldn’t be able to hold onto Phantom’s fave orb for even a second. The moment it leaves his hands and the little you inside freaks out, Phantom is blasting the taker with magic and snatches you back. Quickly soothing you and making sure your orb didn’t get scratched.