Each year I would take off and let go (4,3)

I haven't cracked this one yet — but the brilliant team at Fifteensquared have:

P[er] A[nnum] I’D OFF: ‘let go’: the modern euphemism for ‘fire’.