Running deadcode on large open source Python projects
This page describes a planned experiment that will run deadcode on popular and large open source Python projects. No results have been collected yet. Every result table on this page is a template to be filled after the experiment.
Introduction
The experiment asks a simple question: how does deadcode behave when it is pointed at real, large codebases instead of small examples? The page records the repository selection, the metrics, the measurement procedure, and the reporting format, so the experiment can be repeated and compared later. Nothing on this page is a measured result.
Why run this experiment
The experiment has two goals:
Find deadcode limitations that only appear in realistic code, such as dynamic framework patterns, generated modules, and plugin systems.
Turn useful findings into improvements in deadcode itself and, where a finding is confirmed, in the analysed projects.
The experiment is not meant to do the following:
Automatically fix large amounts of upstream code.
Compare deadcode with other analysers or rank tools against each other.
Experiment questions
Does runtime grow roughly linearly with lines of code, or does it grow faster on very large repositories?
Do findings per thousand lines differ by project type, such as a web framework, a data library, or a plugin-heavy application?
Do dynamic framework patterns produce more false positives than plain library code?
Do runs complete without crashes or unusable output on every candidate repository?
How repositories are selected
Candidates are chosen with the following criteria:
The repository is public.
The licence is permissive or otherwise open enough to read, analyse, and quote small excerpts.
The project supports at least one Python version that deadcode supports (
>=3.10).The project is large or popular enough that a result matters to real users.
The code can be analysed as files without importing the project or running its test suite.
The sample mixes application styles so that size and project type can be told apart.
Size, stars, release age, and Python support are recorded at experiment time together with the pinned commit SHA, not guessed in advance.
Candidate repositories
The starting sample is ten repositories: seven large or very large projects and three smaller controls.
Repository |
Tier |
Why it is in the sample |
Limitations it may expose |
|---|---|---|---|
|
Very large |
Widely used web framework with many internal packages and an app registry. |
Dynamic app loading, settings references, and framework entry points. |
|
Very large |
Large application with many integration modules discovered at runtime. |
Plugin discovery, dynamic imports, and string-based registration. |
|
Very large |
Large orchestration project with operators, plugins, and entry points. |
Metaclasses, entry-point loading, and generated or vendored code. |
|
Large |
Widely used data library with generated APIs and extension modules. |
Generated bindings, re-exports, and code that is hard to resolve statically. |
|
Large |
Widely used machine-learning library with Cython and estimator classes. |
Generated or compiled-adjacent code, inheritance, and dynamic attributes. |
|
Very large |
Large automation project with a plugin loader and many modules. |
Loader-based resolution, string references, and vendored collections. |
|
Large |
Test tool with plugin hooks, fixtures, and dynamic collection. |
Hooks, fixture registration, and code used only through plugins. |
|
Large |
Popular library that builds models and validators at runtime. |
Metaclasses, generated models, and names used only at runtime. |
|
Medium control |
Smaller, well-structured framework with familiar extension points. |
A baseline for comparing framework overhead against the very large projects. |
|
Medium control |
Compact, popular library with a simpler static structure. |
A baseline for checking whether a result scales with size or is project-specific. |
This list is a starting plan. The commit SHA and repository metadata are pinned before measurement, and a candidate is replaced only by recording why it could not be used.
Experiment design
The diagram shows the run order that the measurement procedure will detail.
Choose repositories
|
v
Pin commits and record metadata
|
v
Prepare a clean pinned environment
|
v
Run baseline measurement (neutral working directory)
|
v
Run project-config measurement (repository root)
|
v
Collect runtime, findings, memory, and parse failures
|
v
Review a sample of findings for false positives
|
v
Optionally run fixing on a disposable copy
|
v
Report the results in the template
Metrics
Every metric below is collected for both configuration modes, and every value is reported with the mode it belongs to.
Metric |
Type |
Unit |
How it is collected |
|---|---|---|---|
Wall-clock runtime |
Primary |
seconds |
|
Finding count |
Primary |
findings |
integer from |
Findings per thousand Python lines |
Primary |
findings per 1000 lines |
|
Rule distribution |
Secondary |
findings per |
count finding lines by |
Analysed files and lines |
Secondary |
files, lines |
count |
Parse failures |
Secondary |
count, share of analysed files |
count |
Peak memory |
Secondary |
MiB |
maximum resident set size from |
Exit status |
Secondary |
status code |
|
False-positive rate |
Secondary |
share of reviewed sample |
|
Fix success rate |
Secondary |
share of fixing runs |
on a disposable copy only: share of runs whose |
Wall-clock runtime
Definition: the elapsed time in seconds from process start to process exit for one complete deadcode invocation.
Collection: wrap the command with /usr/bin/time -v, for example /usr/bin/time -v deadcode <repository-path> -v --no-color. Run the command once cold, immediately after boot or after clearing the file cache, and record that value separately. Then run it three times warm and report the median of the three warm values, not their mean. The same commands and the same machine are used for every repository.
Finding count
Definition: the number of findings that deadcode reports after all configured filtering, in one configuration mode.
Collection: run deadcode <repository-path> --count --no-color and read the integer it prints. As a fallback, count the finding lines in normal output with deadcode <repository-path> --no-color | grep -c 'DC[0-9][0-9]'. Both forms must agree. Record which configuration mode produced the count, because an exclusion in [tool.deadcode] changes it.
Findings per thousand Python lines
Definition: the finding count normalised for repository size, so repositories of different sizes can be compared.
Formula: finding_count / analysed_python_lines * 1000, where analysed_python_lines is the line count of the same files that deadcode analysed, not the line count of the whole checkout.
Collection: take the finding count from the previous metric, take the analysed file list from the -v output, and sum those files with wc -l. Report the result with two decimal places and the mode.
Rule distribution
Definition: how the findings are spread across the individual rules, so a single rule cannot silently dominate a result.
Collection: from --no-color output, count the DCxx tokens per finding line, for example with grep -oE 'DC[0-9]{2}' deadcode-output.txt | sort | uniq -c | sort -rn. The codes are printed as DC01 to DC09 and DC11 to DC13; the DC marker used to ignore a finding is not a finding and is not counted. Report absolute counts and shares per mode.
Analysed files and lines
Definition: the number of Python files and Python lines that deadcode actually analysed in one run.
Collection: run deadcode <repository-path> -v --no-color; the verbose log on stderr lists the files to be checked, and the .py paths in that list are the analysed files. Count them, then sum their line counts with wc -l. Compare the file count with the repository’s own Python file count and record every difference, including excluded and unparsable files.
Parse failures
Definition: the number of files that deadcode could not parse and skipped.
Collection: capture stderr and count lines that match Error: Failed to parse <filename> file, ignoring it., for example with grep -c 'Error: Failed to parse' deadcode.stderr. Report the count and its share of analysed files as parse_failures / analysed_files. A parse failure does not change the exit status, so it must be counted from the log rather than inferred from the status.
Peak memory
Definition: the maximum resident set size of the deadcode process during one run.
Collection: read the Maximum resident set size line from the /usr/bin/time -v output of the timed runs. Convert kilobytes to MiB by dividing by 1024. Report the value from the same run whose runtime is reported, and report the cold run separately.
Exit status
Definition: the process status code that deadcode returns for one invocation.
Collection: capture the status immediately with echo $? after the command. The values are 0 when no findings are reported, 1 when findings are reported or an unexpected error occurs, and 2 for invalid usage such as an unknown option. A parse failure alone does not change the status. Record the status for both modes, because findings in either mode are a result.
False-positive rate
Definition: the share of reviewed findings that do not identify unused or dead code, or that the project intentionally keeps and documents.
Formula: false_positive_count / reviewed_count on the reviewed sample.
Collection: draw a stratified sample of up to 30 findings per repository, with the strata being the DCxx rules that occur in that repository. Two reviewers label each finding independently as true positive or false positive, then discuss disagreements and record the agreed labels. The reported value is the agreed false-positive share of the sample, with the sample size, and it applies to the sample, not to all findings.
Fix success rate
Definition: the share of fixing runs that produced a plausible and still valid result.
Collection: run only on a separate disposable copy of the pinned checkout, for example cp -a <repository-path> <copy-path>, then deadcode <copy-path> --fix --no-color and capture the Removed N line. A run counts as successful only when N is greater than zero, every edited file still parses via python -m compileall -q <copy-path>, and a rerun of deadcode <copy-path> --count --no-color reports a lower count. Record every deleted file and every edit that fails to parse. Never run --fix on the recorded checkout, and delete the copy afterwards.
Measurement procedure
The procedure below is the exact run order for every candidate repository. It uses only existing deadcode command line options and [tool.deadcode] settings.
Freeze the deadcode version and commit, the Python version, the operating system, the CPU, the memory, and the exact command lines.
Clone each candidate and check out the pinned commit. Record the commit SHA, licence, release tag, and default branch.
Create a clean environment with the pinned dependencies, install deadcode from the pinned source or wheel, and record
deadcode --version.Run the baseline mode from an empty temporary directory with
deadcode <repository-path> -v --no-color, so no targetpyproject.tomlis read.Run the project-config mode from the repository root with the same arguments, so
[tool.deadcode]applies. Record the effective configuration.For each mode, capture stdout, stderr, and the exit status. Run
--countas a cross-check on the finding count.Time one cold run and three warm runs per mode with
/usr/bin/time -v, and report the median warm runtime and the peak memory.Derive the finding count, rule distribution, analysed files and lines, findings per KLOC, and parse-failure share from the captured outputs.
Draw the false-positive sample, have two reviewers label it, discuss disagreements, and record the agreed labels.
On a separate disposable copy, run
--fix, captureRemoved N, validate the edited files withpython -m compileallorast.parse, and rerun--count.Fill the reporting template, store each run’s full table in its own markdown file in the repository and link it from Table 1, record anomalies and failed candidates, and keep the raw logs with the experiment record.
The two configuration modes are defined as follows. In baseline mode the working directory is an empty temporary directory, so deadcode reads no pyproject.toml from the analysed project. In project-config mode the working directory is the repository root, so [tool.deadcode] is applied exactly as a project user would see it. deadcode reads pyproject.toml only from the current working directory and has no --config option. The file walker includes vendored and generated directories because it has no .gitignore handling, so every exclusion must be explicit and recorded with --exclude or the equivalent [tool.deadcode] key.
Fairness and reproducibility
The following controls keep the comparison fair and the result repeatable:
Tool and repository versions are pinned, and the record names the deadcode version, the deadcode commit, and every repository commit.
All runs happen on one documented machine with the recorded CPU, memory, and operating system.
Every repository is analysed with the same Python version (3.12) and the same deadcode build.
No other workload runs during a timed measurement.
The first run is cold and is reported separately; the reported runtime is the median of the following three warm runs.
The configuration mode is stated for every reported value, and both modes are reported for every repository.
File discovery rules are identical across repositories, with no implicit skips.
Every exclusion is explicit, recorded with its source, and applied in both modes or only in one mode with the reason recorded.
The file walker includes vendored and generated directories, so an exclusion that is not recorded would silently change the analysed file set and the findings per KLOC.
Reporting template
Every table below is empty on purpose. To be completed after the experiment.
Table 1 records one row per candidate repository. Baseline is the neutral-directory mode and Project config is the repository-root mode. A cell that holds two values reports them in that order as baseline / project config, so both modes stay visible in one row. The deadcode version is recorded for every repository because repeated runs with later releases can be compared.
Each run keeps its own full results table in a separate markdown file in this repository, for example docs/experiments/<project>-<commit>.md. The page does not duplicate those tables; the Results file column links to each one.
Repository |
Commit |
Deadcode version |
Runtime (s) baseline / project config |
Findings baseline / project config |
Findings per KLOC baseline / project config |
Peak memory (MiB) baseline / project config |
Parse failures baseline / project config |
Reviewed false-positive rate |
Exit status baseline / project config |
Results file |
|---|---|---|---|---|---|---|---|---|---|---|
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
|
- |
- |
- |
- |
- |
- |
- |
- |
- |
- |
Table 2 records the rule distribution for the experiment as a whole. To be completed after the experiment.
Rule |
Findings |
Share |
|---|---|---|
|
- |
- |
|
- |
- |
|
- |
- |
|
- |
- |
|
- |
- |
|
- |
- |
|
- |
- |
|
- |
- |
|
- |
- |
|
- |
- |
|
- |
- |
|
- |
- |
Results narrative: the experiment will report which repositories completed, which limitations appeared, and how the false-positive sample was judged. To be completed after the experiment.
Threats to validity
Repository selection is purposive rather than random, so the sample describes these projects and does not estimate a population average.
A pinned commit is a snapshot of a moving project, and later commits may behave differently.
Manual labels are subjective, and two reviewers can still share a blind spot.
Wall-clock timing depends on hardware, operating-system state, and cache behaviour, so absolute seconds compare poorly across machines.
--fixresults show that the edited copy still parses; they do not prove that the project’s own tests still pass.The results are descriptive and do not establish a general accuracy rate for deadcode.
How results will be used
Findings become triaged issues for deadcode. Reproducible examples become regression tests. Confirmed false positives become documentation or configuration guidance, such as an ignore marker or a [tool.deadcode] exclusion. Confirmed true positives in a target project become a proposed upstream change after the maintainers have reviewed it.
Out of scope
Comparing deadcode with vulture or any other tool.
Changing deadcode code as part of the experiment.
Running upstream test suites.
Opening upstream pull requests, at least for now.
Publishing raw third-party source or logs that are not licence-compatible.