a regular agent - it's not that he has a problem with Neal, but he's not going to go out of his way for him either

That makes a lot of sense. I enjoy the trope of woobie Neal being abused by an evil handler as much as the next h/c lover, but it's so much more realistic for most agents to feel like, while Neal is of course a human being and they need to be decent to him and make sure he's taken care of if he gets hurt, he's not one of them, not a friend or a colleague. I can't even blame anybody for going that route, no matter how much I love Neal.