PCEval: Physical Computing Capabilities of Large Language Models

PCEval workflow: project descriptions, circuits and code are provided to LLMs; generated artifacts undergo circuit validation and execution-based testing.

“In physical computing, working with circuits is absolutely essential, and that’s just something AI can’t help with. These days, tools like Gemini in Colab can write and even suggest code, which is great. But it’s not like an arm can suddenly pop out and build the circuit for you.”

— Public High School Computer Science Teacher C, Appendix A, Q4

To address this gap, we introduce PCEval (Physical Computing Evaluation), the first benchmark designed to systematically evaluate whether LLMs can reason under physical constraints.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, including software development, education, and technical assistance. However, a fundamental question remains unresolved: can LLMs reason about physical constraints—the rules governing how abstract computations must map onto real-world hardware? This capability is distinct from, and arguably more demanding than, software generation alone, as it requires grounding language understanding in spatial, electrical, and connectivity constraints.

To address this gap, we introduce PCEval (Physical Computing Evaluation), a benchmark and evaluation protocol for physical computing education that enables fully automatic, execution-based assessment of LLMs across both logical and physical implementation artifacts, without human grading. Our evaluation framework assesses LLMs in generating circuits and producing compatible code across varying levels of project complexity.

Through comprehensive testing of 14 leading models, PCEval provides the first reproducible and automatically validated empirical assessment of LLMs’ ability to reason about fundamental hardware implementation constraints within a simulation environment. Our findings reveal that while LLMs perform well in code generation and logical circuit design, they struggle significantly with physical breadboard layout creation, particularly in managing proper pin connections and avoiding circuit errors. PCEval advances our understanding of AI assistance in hardware-dependent computing environments and establishes a foundation for developing more effective tools to support physical computing education.

PCEval Benchmark Design

75Curated projects

Arduino Uno
ESP32Boards

17Evaluated models

300Task instances

Tasks

Five core components: (D)escription, (L)ogical circuit, (P)hysical circuit, (C)ode, (T)est procedure

D, C → L

Logical Circuit Generation

Evaluates ability to infer necessary hardware components and their abstract logical connections based on functional requirements and program logic.

D, C → P

Physical Circuit Generation

Assesses the LLM’s understanding of practical hardware integration, including valid wire routing adhering to breadboard mechanics and physical constraints.

D, L → C

Code Generation from Logical Circuit

Tests the ability to translate a conceptual circuit design into working code.

D, P → C

Code Generation from Physical Circuit

Generate the code that is compatible with this specific physical hardware configuration, including adherence to the explicit pin assignments dictated by the breadboard layout.

Automated Evaluation

Prior efforts on LLMs for electronics relied on labor-intensive manual expert evaluation. Our framework eliminates subjective manual assessment through decomposed tasks and simulation-based validation, enabling reproducible evaluation without domain expert intervention.

We ultimately selected Wokwi for automated evaluation due to its API accessibility and component coverage, while the framework itself remains simulator-agnostic.

Button LED — Success

Button LED — Failure

Quantitative Results

Comparative performance (success rate %) of various LLMs on PCEval benchmark tasks, highlighting differential capabilities across circuit generation and code generation domains.

Success rates for Logical Circuit Generation across all 17 evaluated models. Exact values are available in the data table below.

Success rates for Physical Circuit Generation across all 17 evaluated models. Exact values are available in the data table below.

Success rates (%)

Mean over five independent trials
Model	D, C → L	D, C → P	D, L → C	D, P → C	Overall
Closed Source
GPT 5.6 Sol	59.5	60.0	77.3	69.1	66.5
GPT 5.6 Terra	46.4	40.3	72.5	68.8	57.0
Claude Sonnet 5	67.2	16.0	70.4	72.3	56.5
GPT-4o-mini	50.9	1.3	54.4	53.6	40.1
Gemini-2.0-Flash-Lite	51.2	2.7	56.3	57.3	41.9
Claude Haiku 4.5	58.1	3.7	69.1	64.3	48.8
Gemini-2.0-Flash	60.5	17.9	59.7	55.7	48.5
Gemini-3.1-Flash-Lite	71.5	26.9	72.3	68.8	59.9
GPT-4.1	54.1	16.0	68.8	63.2	50.5
GPT-4o	62.4	28.5	65.3	61.1	54.3
GPT-5-mini	56.5	44.8	70.9	72.5	61.2
o3-mini	65.3	46.4	71.5	71.2	63.6
Open Source
LLaMA 3.1 (8B)	24.3	1.6	29.6	25.9	20.3
DeepSeek-Coder-V2 (16B)	33.3	0.8	34.1	21.1	22.3
Gemma 3 (27B)	52.8	1.6	36.0	37.3	31.9
Phi 4 (14B)	37.1	2.4	49.1	37.9	31.6
Mistral-Small 3 (24B)	51.5	14.1	49.1	35.7	37.6

Qualitative Results

Traffic-light project circuit examples from the paper, comparing successful and failed generated breadboard layouts. — Qualitative examples of LLM-generated physical circuits for a traffic light project (level 4),
illustrating successful and failed attempts with corresponding error analyses.

ESP32 pushbutton-controlled LED: generated circuits from o3-Mini, Claude Haiku 4.5, and Gemini 3.1-Flash-Lite compared with the reference circuit. — This is a ESP32 project that controls an LED with a pushbutton.

BibTeX

@inproceedings{song2026pceval,
  title     = {{PCEval}: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models},
  author    = {Song, Inpyo and Chon, Eunji and Lee, Jangwon},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2601.02404}
}