Why are we talking like the prodigal son got shafted? He never got shafted; didn't his father tell him "Son, thou art ever with me and all that I have is thine"? This is huge because:(1) "Son"--the Prodigal wanted to be a servant and was flummoxed that he got to be a son. Here the father reaffirms to the Other Brother that he is a Son; he's still got that honored role
While I think the Other Brother is maybe exaggerating about never having had a kid with his friends, I'm pretty certain that his father (remember, who's so generous that he would give "bread enough and to spare" to his servants) never let the Other Brother starve. I'm pretty sure he enjoyed the bounty of the fruitful fields he was working, the safety of the house where the party was happening, and the constant support and company of the relations* and servants playing music and dancing inside. Most importantly, he enjoys the company of the Father being "ever with" him.
*I'm betting there are plenty of sisters, cousins and in-laws implied in this story. Maybe the Other Brother even has a wife, or at least reputable options in the neighborhood.