GeoPyEval: An Automated Evaluation Framework for Python Code Generation Capabilities of Large Language Models in Geospatial Domains
Published online on June 15, 2026
Abstract
["Transactions in GIS, Volume 30, Issue 4, June 2026. ", "\nABSTRACT\nDespite the increasing adoption of large language models (LLMs) for Python geospatial code generation, no automated evaluation framework exists for this domain. We propose GeoPyEval, the first function‐level framework in this field, featuring an expert‐in‐the‐loop, automated execution design with three core modules: (a) GeoPyEval‐Bench, constructed from expert‐verified LLM‐generated content, covering 18 mainstream geospatial libraries with 1074 unit test tasks; (b) an expert‐configurable submission program for invoking LLMs to perform code generation and execution; and (c) a judging program for correctness evaluation. We systematically assessed 12 mainstream LLMs available as of November 2025 regarding accuracy, resource consumption, operational efficiency, and error type logs. Results show that GPT‐5 achieved the highest accuracy with a Pass@5 of 64.28%. GeoPyEval extends existing LLM evaluation frameworks to Python geospatial code generation, establishing a novel benchmarking standard for this domain.\n"]