Autonomous vehicles have tremendous social and economic values for society. But before we can see large-scale deployment, its perception systems need to make significant progress in robustness and generalization capability. To address these challenges, we bring context, the semantic meaning into the robot perception. Our approach is to conduct research on our full-scale testing vehicles, our living laboratory. We evaluate the state-of-the-art algorithms, search for and address the challenges we found in real-world deployment. Such challenges include long term tracking and prediction of road users, dynamic scene modeling and dynamic planning.