Revert crash fix when removing player layer observer

Couldn't figure out how this was happening, so I don't want to hide it with try/catch. Let's get it fixed instead.