Abstract

To investigate the genotype-phenotype link in a polyclonal cancer cell population, here we introduce evolutionary game theory into our previously developed agent-based brain tumor model. We model the heterogeneous cell population as a mixture of two distinct genotypes: the more proliferative Type A and the more migratory Type B. Our agent-based simulations reveal a phase transition in the tumor's velocity of spatial expansion linking the tumor fitness to genotypic composition. Specifically, velocity initially falls as rising payoffs reward the interactions among the more stationary Type A cells, but unexpectedly accelerates again when these A-A payoffs increase even further. At this latter accelerating stage, fewer migratory Type B cells appear to confer a competitive advantage in terms of the tumor's spatial aggression over the overall numerically dominating Type A cells, which in turn leads to an acceleration of the overall tumor dynamics while its surface roughness declines. We discuss potential implications of our findings for cancer research.