DEP 1 - More accurate usage detection

Status: Draft

Type: Feature

Created: 2024-01-14

Problem statement

Usage detection can be more accurate. It is common in other unused code detection tools to simply construct sets of names used in definitions and names used in usage expressions. And reporting the names, which are in defined set, but not in usage one.

In larger projects cases when names overlap are quite common. In this example:

class Foo:
    def validate(self):
        pass

class Bar:
    def validate(self):
        pass

Foo().validate()

validate name would not be reported if this simple strategy is being used. However, it is possible to track the types of definitions and usages and more accurately identify the usages. This DEP 1 proposal specifies the strategy, which could be used to more accurately track the types of variables.

Considered alternatives

  • Keep the current name-based matching. This is simple, but cannot distinguish definitions that share a name, so a name is reported only when no definition with that name is used.

  • Track definitions and usages by their scope strings. This separates definitions in different modules and classes, but still treats classes, functions, and variables with the same scope as equivalent.

  • Track the type of every scope part as well as the scope string. This keeps the scope information from the previous option and resolves overlapping names by their kind.

Chosen solution

Track the types of definitions and usages as well as their scopes, so that overlapping names can be matched to the definition that the expression actually refers to.

Scope construction for definitions:

Dot-separated scope name is being used to identify a scope in each line.

# foo.py         # foo
class Bar:       # foo.Bar
    def spam():  # foo.Bar.spam
        pass     # foo.Bar.spam

If the foo.py file is not on the working path, then its scope is being prefixed with dot separated package names. For example, if the foo.py is in ham.eggs package, then the scope of spam method will be: ham.eggs.foo.Bar.spam.

Matching defined type with a type in an expression

Lets say we have a Bar type definition and an expression which uses it:

# foo.py         # SCOPE
class Bar:       # foo.Bar
    def spam():  # foo.Bar.spam
        pass     # foo.Bar.spam
Bar().spam()     # foo, TYPE: Bar

On the expression line Bar().spam() the scope is foo, the identified type name is Bar. This type Bar will be used to search for a type definition (CodeItem instance) in a namespace foo and usages will be marked on it as well as usages of related attributes/methods will be associated with that type.

Providing only a part of scope via options is fine

All of these scopes will match the spam method: ham.eggs.foo.Bar.spam foo.Bar.spam Bar.spam spam

The less specific the scope is the more cases it will match, i.e. spam scope would also match a variable named spam in any scope as well. In some sence scopes like spam`` have wild cards *.spam` matching any scope prefixes.

Identifying types of scope parts

When creating the scope parts, each part could also have the type (e.g. pacakge, module, class, method, variable) associated with it. When a usage expression is being detected its type could be searched by using types of scope part, instead of simply comparing scope strings.

For example this code snippet contains two different objects, which could be matched using a generic foo.bar scope:

def foo:
    bar = 1

class foo:
    def bar(self):
        pass

foo().bar = 1

Deadcode could internally track the type of each scope part and when an expression is being detected, the defined type could be searched by taking the types of scope parts, not only the scope string. For example, using a special notation like >foo%bar and #foo>bar for scopes could be used for the above example to accurately identify definitions.

User could also provide precise types of scope parts by using a different separator instead of .. These separators could be used for scope part separation:

  • . - means any type of scope

  • / - package or module scope

  • # - class scope

  • > - function or method scope

  • % - variable or variable attribute

For example, user could provide this ham.eggs.foo.Bar.spam path as well as a more specific one ham/eggs/foo#Bar>spam to exactly match the types of scope parts.

Type tracking for method arguments and returned values

Tracking type of arguments

When argument is being passed into a function/method the type remains the same, but the variable name might change, or it might be put into a container like tuple or dictionary. Deadcode will attempt to track the types of function/method parameters, however in some cases the type will be lost and deadcode will fallback to a generic name matching strategy.

In this example:

class Foo:
    def bar(self):
        pass

def eggs(ham):
    ham.bar()

spam = Foo()
eggs(spam)

Deadcode will be able to accurately detect that type of ham is Foo.

Tracking types of returned values

It might be hard to track exact types of variables, for example:

clas Eggs:
    pass

class Bar:
    def spam():
        return Eggs()

variable = Bar().spam()
print(variable)

Parsing the returned type of Bar.spam is complicated. In some cases, the returned type might only be determined dynamically during a runtime and it might depend on method’s implementation details. Hence, in some cases the types won’t be identified due to runtime not being available during static code analysis.

The Deadcode policy on this is that when a type is being lost due to inability to accurately identify it.

In such cases the Deadcode will loose a way to accurately identify the type of variables/attributes. Hence a generic name matching will be used instead in these cases, just like vulture does. If more than one definition with the same name is detected the warning should be issues (if enough verbosity is enabled). In addition, type hints could be used to try to detect the type more easily in such cases.