The self-formation of social aggregations is simulated using idealized mathematical and numerical models. When the mating probability is density-dependent, social behavior increases reproductive success, although this is offset by enhanced competition for food resources within group members. The costs and benefits of aggregation are implicitly included in the predator-prey interactions, as populations with different strategies compete for a common prey. The outcome of the competition depends on environmental conditions, population dynamics, and the type of behavior; it can be determined by formulating coupled partial differential equations for the densities of the different populations and their prey, and integrating numerically. Weakly nonlinear equations are used to make analytical predictions regarding the success of various strategies. Evolutionary stable strategies can be identified if the spatial scale of the aggregations is fixed.