Pastor, God called you into this work, did He? Then, do yourself a favor by keeping your eyes on Him. If you begin to fixate on the congregation for your identity and your purpose in life–if pastoring that church is your life–nothing good is going to come of it.
Sooner or later, you are going to have to walk away from the church you are presently serving. You may retire, be called to another church, get fired, or transition into another kind of ministry (denominational service, etc.). In any case, you will be required to relinquish every aspect of that ministry: to move out of the office and take your name off the door, as well as give up every phase of leadership, every perk and benefit, and anything else that comes with your pastoral position.
It would be nice if you could go quietly.
Later today, some of us are meeting with a pastor–for the second time–to help his church find a resolution to a frustrating situation with a disarmingly simple remedy. All the turmoil inside the congregation would disappear in a heartbeat if the pastor would recognize that, now that he is in his mid-seventies, he needs to retire. The congregation wants him to–even those who love him–and he needs to do so.
Turning loose and walking away. Giving the church back to Jesus.