Phog: You have to be level 14, have beaten chapter 4, have taken the basic mission "Backup Request" (does not have to be completed), and then you take the affinity mission "Froggy Dilemma" (this is right below the administrative district, in the residential district).Alexa: You have to have beaten chapter 4, have taken the basic mission "Guinea Pig Wanted" (you do not have to complete it), and then take the affinity mission "Weaponised" (this is in the test hangar in the industrial district).Yelv: You have to be level 15, have beaten chapter 3, and complete the affinity mission "Meeting Yelv" (this is in the commercial district).H.B.: You have to be level 12, have beaten chapter 3, and complete the affinity mission "A New Rival" (this is in the administrative district).Almost all of them require you do affinity missions (plus, there are some other pre-requisites like level and story chapter). This gets you a pretty good sized party, but there are plenty more characters to unlock.
A handful of characters are obtained via story, namely these: