Although recent decades have witnessed significant success in deploying robots and autonomous systems in laboratory developments, manufacturing plants, transportation, and home applications, the systems lack the intelligence and robustness to operate reliably in unstructured environments and under adverse conditions. When humans perceive and navigate complex environments, we identify various places and objects, i.e., semantics, and infer their properties, physics, and relationships from semantics based on prior knowledge and experience. However, it is challenging to enable robots and autonomous systems to have such a high-level understanding of their surroundings, given only noisy sensor measurements and limited onboard computing resources, especially for lightweight unmanned aerial systems. In this talk, I will focus on a semantic perception system I developed for field robots that uses semantics as a pivot to achieve high-level scene understanding and reliable state estimation for planning and control. In addition to bringing human-level semantics to robot perception in a 3D continuous semantic map representation for task planning, the system also reasons robot-specific properties of the environment to assist more sophisticated robot behavior planning. Effective learning methods to enable semantics acquisition under adverse sensing conditions will also be discussed. Future research aims to extend current semantic perception to dynamics-aware robot perception and increase the understanding of environmental dynamics for autonomous systems in the wild.