Abstract
A hierarchical repository-analysis pipeline that filters a codebase, summarizes it at file, module, and repository granularity, and generates a README draft using a local Qwen2.5-Coder-7B model with GPT as a comparison baseline. The core engineering problem is keeping context within a 7B model's window while preserving enough structural detail for useful documentation.
1. What This Is
The project grew from 2024–2025 thesis work on automated repository understanding. Two artifacts exist: the broader research implementation (local Qwen2.5-Coder-7B served via LM Studio/llama.cpp, plus GPT API baselines) and a smaller public GitHub tool that enumerates repositories, summarizes Python source, and writes generated documentation back through the GitHub API.
The public repository is the practical slice; the thesis work added the evaluation layer comparing local-model and GPT outputs on correctness, completeness, hallucination control, latency, and privacy trade-offs.
2. How It Works
The pipeline is strictly hierarchical: each level consumes the outputs of the level below, so no single prompt ever receives the full repository. Filtering happens before any LLM call to concentrate context on source and project-defining files.
| # | Stage | Input | Tool | Output |
|---|---|---|---|---|
| 01 | Collect | Repository URL | GitHub API + httpx | Full file tree |
| 02 | Filter | File tree | Python rules | Source, config, docs only |
| 03 | File summary | Individual source files | Qwen2.5-Coder-7B / GPT | Per-file summaries |
| 04 | Module summary | File summaries | LLM synthesis | Module-level descriptions |
| 05 | Repo summary | Module summaries + metadata | LLM synthesis | Repository understanding |
| 06 | README draft | Repo summary | LLM | Markdown README |
| 07 | Publish (optional) | README draft | GitHub API | Updated repo docs |
3. Implementation Notes
3.1 Context budgeting and hierarchy
A 7B model has a hard context window. Concatenating an entire repository into one prompt produces unreliable, hallucination-prone output. The three-level hierarchy (file → module → repo) keeps each LLM call within budget while preserving structural relationships. Filtering of dependency folders, caches, binaries, and oversized assets happens before any model call so the context is concentrated on code that actually defines the project.
3.2 Local model serving
Qwen2.5-Coder-7B is served through an OpenAI-compatible endpoint (LM Studio or llama.cpp-style). This keeps analysis private and free at inference time. A parallel GPT API path exists for the thesis baseline comparison, letting me measure the quality gap between local and cloud models on the same inputs.
3.3 Prompt structure
Separate prompts exist for file-level code summarization, module-level synthesis, and final README generation. Each prompt is structured around software-documentation tasks rather than generic chat, with explicit instructions to preserve implementation details and avoid unsupported claims.
3.4 GitHub automation
The public tool uses httpx against the GitHub REST API for repository discovery, recursive file listing, content retrieval, and writing generated documentation back to the target repository. This closes the loop from analysis to published output without manual intervention.
4. Constraints
-
7B context ceiling
The local model's window forces aggressive filtering. Large monorepos or repos with heavy generated code may still exceed budget even after filtering, degrading summary quality.
-
No automated quality metrics
Thesis evaluation was qualitative (correctness, completeness, hallucination). There is no reproducible scoring pipeline, so regressions in prompt or model changes are hard to detect.
-
Python-centric public tool
The GitHub-facing implementation targets Python source. Multi-language repos (TypeScript, Go, Rust) are not handled by the filtering or prompting layer in the public version.
-
Full re-analysis every run
There is no caching or delta detection. Each invocation re-summarizes the entire repository, which is expensive on the local model and wasteful for small commits.
5. Next
- a. Add an LLM-as-judge or BLEU/ROUGE scoring step so summary quality is measurable across model and prompt changes.
- b. Extend the filtering and prompting layer to handle TypeScript, Go, and Rust repositories in the public tool.
- c. Implement incremental summarization: hash file contents, skip unchanged files, and merge new summaries into the existing module and repo hierarchy.
- d. Package the pipeline as a GitHub Action so a README can be regenerated on a schedule or on push without manual invocation.
— end of report —