Accurate prediction of protein properties that influence cellular functions is crucial for drug design, disease research, and guiding biological wet-lab experiments. Previous methods primarily relied on physicochemical property analysis and homologous sequence alignment, lacking end-to-end solutions. In recent years, Protein Language Models (PLMs) pretrained on large-scale residue sequences have shown impressive results in protein engineering. However, protein functions are highly dependent on complex spatial structures. Fine-tuning PLMs either by relying solely on sequence information or by incorporating structure-aware features within language modeling methods has not led to substantial improvements in predictive performance. To address this, we propose a novel Graph with Residues (GwR)-based cross-modal framework. GwR employs a Layer-Aggregated Graph Convolutional Network (LA-GCN) and a Geometric Vector Perceptron-Graph Neural Network (GVP-GNN) to perform representation learning on two complementary residue graphs: one is based on PLMs and self-attention mechanisms to capture semantic features and dynamic residue associations from sequences, and the other incorporates structure-aware sequences and spatial topology to describe the structural characteristics of proteins. We apply GwR to four protein property prediction tasks, including subcellular localization, solubility, metal ion binding, and thermal stability and conduct extensive comparisons with PLMs. Experimental results demonstrate that GwR consistently outperforms existing methods in terms of both predictive performance and training efficiency. Furthermore, GwR exhibits superior or comparable performance when evaluated against multiple state-of-the-art deep learning models and parameter-efficient fine-tuning strategies.