He left his Dominican Republic Winter League team, Escogido, after hitting .069 in nine games.
A team spokesman said Saturday night Brown left because he was tired and sluggish and they want him ready for Spring Training.
Depending how Brown performs this spring, he could help replace Jayson Werth in right field, if Werth signs elsewhere as a free agent. But the organization has said Brown must earn a job.
Phillies general manager Ruben Amaro Jr. said earlier this week he was not concerned with Brown's performance in winter ball because he had not played much late in the season and 20-plus at-bats in winter ball were not enough to make an evaluation.
Todd Zolecki is a reporter for MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.