Structuring Python Projects and Managing Application Dependencies

Course: Python Training

Topic: Structuring Python Projects and Managing Application Dependencies

Course Link: Python Training Course

Demo Link: Register for a Course Demo

Website: JustAcademy

1. Introduction

Structuring Python projects and managing application dependencies are essential skills for building reliable, scalable, and maintainable software. A small Python script may work well as a single file, but larger applications often contain many modules, packages, configuration files, tests, documentation files, and external libraries.

A clear project structure makes it easier to locate code, reuse functionality, collaborate with other developers, and test individual components. Dependency management ensures that the application uses the required libraries and compatible versions across development, testing, and production environments.

Python provides tools and conventions such as packages, virtual environments, pip, requirements.txt, and pyproject.toml to help organize projects and manage dependencies.

2. Learning Objectives

• Understand the importance of a well-structured Python project.
• Organize Python modules, packages, tests, and configuration files.
• Understand the purpose of virtual environments.
• Install and manage third-party libraries using pip.
• Create and use a requirements.txt file.
• Understand the role of pyproject.toml.
• Differentiate direct and transitive dependencies.
• Manage dependency versions and reduce compatibility issues.
• Use environment variables for configuration and secrets.
• Apply best practices for project maintenance and deployment.

3. What Is Python Project Structure?

Project structure refers to the organization of files and directories within a software application. A good structure separates different responsibilities instead of placing all code in one large file.

For example, an application may contain separate directories for business logic, database operations, automated tests, documentation, and configuration.

Benefits of a Good Project Structure

• Maintainability: Developers can locate and update code more easily.
• Reusability: Functions and classes can be imported into multiple modules.
• Testability: Individual components can be tested independently.
• Scalability: New features can be added without making the codebase unnecessarily complicated.
• Collaboration: Team members can work on separate components more effectively.
• Deployment: Applications are easier to package, configure, and release.
• Dependency isolation: Project libraries can be separated from those of other applications.

4. Basic Python Project Structure

A simple Python application might initially contain only a few files:

calculator_project/
├── main.py
├── calculator.py
└── README.md

Here, main.py starts the application, calculator.py contains reusable calculation functions, and README.md explains how to use the project.

Example: calculator.py

def add(a, b):
    return a + b

def subtract(a, b):
    return a - b

Example: main.py

from calculator import add, subtract

print("Addition:", add(10, 5))
print("Subtraction:", subtract(10, 5))

Output:

Addition: 15
Subtraction: 5

This structure is appropriate for a small program. As the application grows, its code can be divided into packages and additional directories.

5. Recommended Structure for a Larger Python Application

A larger application benefits from separating application code, tests, documentation, and project configuration.

task_manager/
├── pyproject.toml
├── README.md
├── .gitignore
├── .env.example
├── src/
│   └── task_manager/
│       ├── __init__.py
│       ├── main.py
│       ├── models.py
│       ├── services.py
│       └── config.py
├── tests/
│   ├── test_models.py
│   └── test_services.py
└── docs/
    └── usage.md

This is a common structure, not a mandatory standard. Smaller projects may use a simpler layout, while web applications, data science projects, and command-line tools may require different directories.

6. Organizing Code into Modules and Packages

A module is generally a Python file containing functions, classes, and variables. A package groups related modules in a directory and can contain subpackages.

For example, an inventory application could be organized as follows:

inventory_app/
├── main.py
└── inventory/
    ├── __init__.py
    ├── products.py
    ├── stock.py
    └── reports.py

The modules can have distinct responsibilities:

• products.py manages product information.
• stock.py handles stock quantities and updates.
• reports.py generates inventory summaries.
• __init__.py can expose selected package functionality.

Example: Importing from a Package

# inventory/products.py

def get_product_name(product_id):
    products = {
        1: "Keyboard",
        2: "Mouse"
    }
    return products.get(product_id, "Unknown product")

# main.py

from inventory.products import get_product_name

print(get_product_name(1))

Output:

Keyboard

Separating functionality into modules makes the application easier to understand and maintain. Avoid splitting code into many tiny modules unless the separation provides a clear benefit.

7. What Are Application Dependencies?

A dependency is an external library, package, service, or component that an application requires to perform its tasks. In Python projects, the term commonly refers to installable packages.

For example, a web application might use Flask, a data analysis application might use pandas, and an HTTP client might use requests.

Types of Dependencies

8. Why Dependency Management Is Important

Applications often rely on specific library versions. If a dependency changes its API or introduces incompatible behavior, the application may fail even when its own source code has not changed.

Dependency management helps developers:

• Install required libraries consistently.
• Recreate a development environment on another computer.
• Reduce conflicts between package versions.
• Separate development tools from runtime requirements.
• Prepare reliable testing and deployment environments.
• Review dependency changes before upgrading.
• Identify outdated or vulnerable packages.

9. Installing Dependencies with pip

pip is Python's commonly used package installer. It installs packages from configured package indexes and can also install packages from other supported sources.

Check Python and pip

python --version
python -m pip --version

On systems where the Python command is named python3, use:

python3 --version
python3 -m pip --version

Using python -m pip helps ensure that pip runs under the selected Python interpreter.

Install a Package

python -m pip install requests

Install a Specific Version

python -m pip install "requests==2.32.3"

This example requests a particular version. In an actual project, choose a version compatible with your supported Python version and other dependencies rather than copying a version without checking compatibility.

View Installed Packages

python -m pip list

Inspect Package Details

python -m pip show requests

Uninstall a Package

python -m pip uninstall requests

Before uninstalling a package, check whether other application components rely on it.

10. Understanding Virtual Environments

A virtual environment provides an isolated Python environment for a project. It keeps the project's installed packages separate from the global Python environment and other projects.

For example, Project A might require one version of a library, while Project B requires a different version. Using separate virtual environments reduces conflicts between those projects.

Create a Virtual Environment

From the project directory, run:

python -m venv .venv

The command creates a virtual environment in a directory named .venv.

Activate on Windows Command Prompt

.venv\Scripts\activate.bat

Activate on Windows PowerShell

.\.venv\Scripts\Activate.ps1

Activate on macOS or Linux

source .venv/bin/activate

After activation, install the packages required by the project:

python -m pip install requests

When the environment is active, the shell often displays its name in the prompt. You can confirm the selected interpreter using:

python -c "import sys; print(sys.executable)"

Deactivate the Environment

deactivate

Deactivation returns the shell to its previous environment context. It does not delete the virtual environment or uninstall its packages.

11. Managing Dependencies with requirements.txt

A requirements.txt file records package requirements that can be installed with pip. It is a widely used way to communicate an application's dependencies to other developers and deployment systems.

Example requirements.txt

requests==2.32.3
pytest==8.3.5

These are illustrative version pins; verify that the chosen versions support your Python version and project requirements before using them.

Install Dependencies from the File

python -m pip install -r requirements.txt

Pip reads the file and installs the listed requirements, along with their required dependencies.

Generate a Snapshot of the Current Environment

python -m pip freeze > requirements.txt

The command records installed distributions and versions in a requirements-style format. It may include packages that are not direct application dependencies, so review the generated file before treating it as the project's intended dependency specification.

When to Use requirements.txt

• When a simple project needs an easy way to install its dependencies.
• When a team uses pip-based environment setup.
• When deployment instructions expect a requirements file.
• When reproducing an environment from a reviewed package list.

12. Dependency Version Specifiers

Dependency declarations can specify which package versions are acceptable. Selecting appropriate version constraints helps balance compatibility, maintenance, and access to updates.

For example, requests>=2.30,<3.0 permits versions starting at 2.30 but excludes version 3.0 and later. A range is not a guarantee that every permitted version works with your application; testing is still necessary.

13. Understanding pyproject.toml

pyproject.toml is a standard configuration file used by Python packaging tools and many development tools. It can contain project metadata, build-system configuration, dependencies, optional dependency groups, and tool settings.

For new Python projects, it is often a useful central location for project configuration. Its exact contents depend on the packaging backend and tools being used.

Example pyproject.toml

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "task-manager"
version = "0.1.0"
description = "A simple task management application"
requires-python = ">=3.10"
dependencies = [
    "requests>=2.30,<3.0"
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "ruff>=0.5"
]

This example uses setuptools as the build backend. The project's runtime dependency is declared under [project].dependencies, while tools useful during development are placed in an optional dependency group named dev.

The version constraints are illustrative and should be reviewed for the target Python version and application.

Install a Project with Development Dependencies

For a project using the configuration above, install it in editable mode with its development extras:

python -m pip install -e ".[dev]"

Editable installation makes the project importable while allowing changes to the source tree to be used without reinstalling the project after every code edit. It is especially useful during development.

14. requirements.txt vs. pyproject.toml

These files can coexist. A project may declare its package metadata in pyproject.toml and maintain a requirements file for a specific deployment workflow. Avoid maintaining conflicting dependency definitions without a clear reason.

15. Direct and Transitive Dependencies

A direct dependency is explicitly required by your application. A transitive dependency is installed because another package requires it.

Suppose your code imports requests. Your application has a direct dependency on requests. Requests may rely on other packages to provide functionality; those packages are transitive dependencies from your application's perspective unless your code also requires them directly.

Why the Difference Matters

• Declare packages your application uses directly as direct dependencies.
• Allow the package installer to resolve transitive dependencies according to the dependency metadata and version constraints.
• Review the complete installed environment when investigating conflicts or vulnerabilities.
• Avoid importing a transitive dependency directly without declaring it if your application relies on it.

16. Dependency Resolution and Compatibility

Dependency resolution is the process of selecting package versions that satisfy declared constraints and dependency requirements. Two packages may require incompatible versions of the same dependency, preventing a valid environment from being created.

Example of a Potential Conflict

library_a requires shared-library < 2.0
library_b requires shared-library >= 2.0

These constraints do not overlap, so a resolver cannot choose a version that satisfies both requirements.

Ways to Handle Dependency Conflicts

• Check the full error message from pip.
• Review the dependency requirements of the conflicting packages.
• Upgrade or replace a package when a compatible release is available.
• Use compatible package versions that satisfy all requirements.
• Test the resulting environment before deployment.
• Do not force-install incompatible packages simply to bypass a resolver error.

17. Using Environment Variables for Configuration

Application configuration often changes between development, testing, and production. Environment variables allow configuration values to be supplied without hardcoding them into source files.

Common examples include database URLs, application mode, log levels, and API credentials.

Read an Environment Variable

import os

app_environment = os.getenv("APP_ENV", "development")
debug_mode = os.getenv("DEBUG", "false").lower() == "true"

print("Environment:", app_environment)
print("Debug mode:", debug_mode)

The default value is used when the variable is not set. Notice that environment variable values are strings, so Boolean values need to be parsed explicitly.

Example .env.example

APP_ENV=development
DEBUG=false
DATABASE_URL=replace_with_your_local_database_url

This example documents the expected variables. It is not a file containing actual secrets. If a project uses a library to load .env files, declare that library as a dependency and configure loading intentionally.

Configuration Security Best Practices

• Never commit real API keys, passwords, or production credentials to source control.
• Add local secret files to .gitignore where appropriate.
• Provide safe placeholders in .env.example.
• Use the deployment platform's secret-management features for production credentials.
• Validate required configuration values at application startup.

18. Testing Dependencies and Application Code

Testing verifies that the application behaves correctly and that changes do not introduce unexpected errors. A well-organized project separates test dependencies from runtime dependencies whenever practical.

Install pytest

python -m pip install pytest

Example Application Function

# src/task_manager/services.py

def calculate_total(prices):
    if any(price < 0 for price in prices):
        raise ValueError("Prices cannot be negative")
    return sum(prices)

Example Test

# tests/test_services.py

import pytest
from task_manager.services import calculate_total

def test_calculate_total():
    assert calculate_total([10, 20, 30]) == 60

def test_calculate_total_empty_list():
    assert calculate_total([]) == 0

def test_negative_price_raises_error():
    with pytest.raises(ValueError):
        calculate_total([10, -5])

Run the tests from the project root after installing the application and test dependencies:

python -m pytest

For a src-layout project, installing the project in editable mode during development helps ensure the package can be imported correctly:

python -m pip install -e ".[dev]"

This command assumes the project has a valid pyproject.toml with a dev optional dependency group and a configured build backend.

19. Using .gitignore for Project Maintenance

Git is commonly used to track source code and collaborate with other developers. A .gitignore file prevents specified local or generated files from being added to version control.

Example .gitignore

# Virtual environments
.venv/
venv/

# Python cache files
__pycache__/
*.py[cod]

# Test and coverage artifacts
.pytest_cache/
.coverage
htmlcov/

# Local environment files
.env
.env.*

# Allow the example configuration
!.env.example

# Build artifacts
build/
dist/
*.egg-info/

Review ignore patterns before using them. The file should exclude generated files and local secrets without accidentally excluding important source code, required configuration templates, or project documentation.

20. Managing Dependencies Across Development and Production

Development and production environments may need different packages. Production usually requires only what is needed to run the application, while development may also require testing, formatting, linting, and debugging tools.

In a project using pyproject.toml, optional dependency groups can help separate these requirements.

[project]
name = "task-manager"
version = "0.1.0"
dependencies = [
    "requests>=2.30,<3.0"
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "ruff>=0.5"
]

Install only the runtime dependencies when that is appropriate for your deployment workflow. For development, install the project with the development extras:

python -m pip install -e ".[dev]"

Optional dependency groups do not automatically create separate environments. Use the installation commands and deployment configuration that match the project's requirements.

21. Updating and Auditing Dependencies

Dependencies need regular maintenance because new releases can provide bug fixes, performance improvements, compatibility updates, and security fixes. However, upgrading a package can also introduce breaking changes.

Inspect Installed Packages

python -m pip list
python -m pip show requests

Check for Dependency Conflicts

python -m pip check

This command checks whether installed packages have compatible declared requirements. It does not guarantee that the application has no bugs or security vulnerabilities.

Upgrade a Package Deliberately

python -m pip install --upgrade requests

After upgrading, run the test suite and review any dependency changes before deploying. Avoid automatically upgrading every production dependency without testing.

Dependency Maintenance Checklist

• Review package release notes before significant upgrades.
• Check supported Python versions and compatibility requirements.
• Run automated tests after dependency changes.
• Review dependency vulnerabilities using appropriate security tools.
• Keep dependency declarations and deployment instructions synchronized.
• Remove unused direct dependencies when it is safe to do so.

22. Reproducible Environments and Lock Files

A reproducible environment is one that can be recreated reliably using documented dependency information. Reproducibility is important when moving a project between computers, CI systems, and production servers.

A requirements file can pin exact versions, while project metadata in pyproject.toml can declare supported version ranges. Some dependency-management tools additionally create lock files that record a resolved set of package versions and, depending on the tool, other installation details.

Recommended Approach

1. Declare the application's direct dependencies clearly.
2. Specify a supported Python version range.
3. Use a virtual environment for isolated development.
4. Choose an appropriate dependency-management and locking workflow.
5. Recreate the environment from the project's declared files in a clean environment.
6. Run tests before accepting the environment for deployment.

Important: A lock file and a requirements file are not automatically interchangeable. Follow the format and workflow expected by the package manager used in the project.

23. Troubleshooting Common Dependency Problems

24. Practical Project: Build a Task Manager Application

This practical project combines package structure, dependency management, testing, and configuration in a small Python application.

Step 1: Create the Project Structure

task_manager/
├── pyproject.toml
├── README.md
├── .gitignore
├── src/
│   └── task_manager/
│       ├── __init__.py
│       ├── services.py
│       └── main.py
└── tests/
    └── test_services.py

Step 2: Configure pyproject.toml

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "task-manager"
version = "0.1.0"
description = "A simple Python task manager"
requires-python = ">=3.10"
dependencies = []

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "ruff>=0.5"
]

[tool.setuptools.packages.find]
where = ["src"]

This configuration defines the project metadata, build backend, supported Python version, development tools, and location of the application package.

Step 3: Create services.py

def add_task(tasks, title):
    title = title.strip()

    if not title:
        raise ValueError("Task title cannot be empty")

    task = {
        "id": len(tasks) + 1,
        "title": title,
        "completed": False
    }

    tasks.append(task)
    return task

def list_tasks(tasks):
    return tasks

Step 4: Create __init__.py

from .services import add_task, list_tasks

__all__ = ["add_task", "list_tasks"]

Step 5: Create main.py

from task_manager import add_task, list_tasks

def main():
    tasks = []

    add_task(tasks, "Learn Python packages")
    add_task(tasks, "Practice dependency management")

    for task in list_tasks(tasks):
        print(task)

if __name__ == "__main__":
    main()

Step 6: Create test_services.py

import pytest
from task_manager import add_task, list_tasks

def test_add_task():
    tasks = []
    task = add_task(tasks, "Learn Python")

    assert task["id"] == 1
    assert task["title"] == "Learn Python"
    assert task["completed"] is False

def test_empty_title_is_rejected():
    with pytest.raises(ValueError):
        add_task([], "   ")

def test_list_tasks():
    tasks = []
    add_task(tasks, "Write tests")

    assert len(list_tasks(tasks)) == 1

Step 7: Set Up the Environment

python -m venv .venv

Activate the environment using the instructions for your operating system, then install the project and its development tools:

python -m pip install -e ".[dev]"

Step 8: Run the Application and Tests

python -m task_manager.main
python -m pytest
python -m pip check

The first command requires the package to have a main.py module as shown above and to be installed or otherwise importable from the selected environment. The test command checks the application behavior, and the final command checks installed dependency requirements.

Expected Learning Outcomes

• Organize a project using the src layout.
• Expose functions through a package's __init__.py.
• Declare project metadata and optional development dependencies.
• Install the project in editable mode.
• Write and run automated tests.
• Check the installed environment for dependency conflicts.

25. Best Practices for Structuring Python Projects

• Start with a simple structure and add complexity only when needed.
• Give modules and packages clear, descriptive names.
• Separate business logic from command-line entry points and configuration.
• Keep tests separate from application code.
• Use a virtual environment for each project.
• Declare direct dependencies instead of relying on accidental installations.
• Choose appropriate version constraints and test dependency upgrades.
• Use pyproject.toml for standardized project metadata and tool configuration.
• Use requirements.txt when it fits the project's installation workflow.
• Keep credentials out of source control.
• Document setup commands in the README.
• Test installation from a clean environment before deployment.

26. Interview Questions and Answers

Q1. What is a Python project structure?

Answer: It is the organization of a project's source files, packages, tests, configuration, documentation, and other resources into directories with clear responsibilities.

Q2. What is a dependency in Python?

Answer: A dependency is a package or component that an application relies on to provide required functionality.

Q3. Why should we use virtual environments?

Answer: Virtual environments isolate project dependencies so that different projects can use different packages and versions without unnecessarily interfering with one another.

Q4. What is pip?

Answer: Pip is a package installer used to install and manage Python packages from configured package indexes and other supported sources.

Q5. What is requirements.txt?

Answer: It is a text file commonly used to record package requirements for installation with pip.

Q6. What is pyproject.toml?

Answer: It is a standardized TOML configuration file used for Python project metadata, build-system settings, dependencies, and configuration for development tools.

Q7. What is the difference between direct and transitive dependencies?

Answer: Direct dependencies are explicitly required by the application. Transitive dependencies are required by other packages that the application installs.

Q8. What is the purpose of a virtual environment activation command?

Answer: It adjusts the shell environment so that Python and package commands use the virtual environment's interpreter and installed packages.

Q9. What is an editable installation?

Answer: An editable installation, commonly created with python -m pip install -e ., makes a local project importable while allowing source changes to be used during development without reinstalling after every edit.

Q10. How can you check for installed dependency conflicts?

Answer: Run python -m pip check to check whether installed packages have compatible declared requirements.

Q11. Why is dependency version pinning useful?

Answer: Version pinning can improve reproducibility by specifying the package versions expected in an environment. It should be paired with a sensible update and testing process.

Q12. What is the difference between runtime and development dependencies?

Answer: Runtime dependencies are needed to run the application, while development dependencies support tasks such as testing, formatting, linting, and debugging.

Q13. Why should secrets not be stored in source code?

Answer: Committed credentials can be exposed to other users and systems. Environment variables and appropriate secret-management tools help keep sensitive configuration separate from application code.

Q14. How do you troubleshoot a ModuleNotFoundError?

Answer: Verify the import name, active Python interpreter, virtual environment, package installation, project layout, and module search path.

Q15. How do you make a Python project easier to maintain?

Answer: Use clear module boundaries, documented setup instructions, isolated dependencies, automated tests, version control, and a consistent configuration workflow.

27. Quick Revision Checklist

• A project structure separates code and supporting resources by responsibility.
• Modules and packages help organize reusable Python functionality.
• Dependencies are packages and components required by an application.
• pip installs and manages Python packages.
• venv creates isolated Python environments.
• requirements.txt records requirements for pip-based workflows.
• pyproject.toml defines project metadata and packaging configuration.
• Direct dependencies should be declared explicitly.
• Dependency versions and compatibility should be tested.
• Environment variables can separate configuration from source code.
• .gitignore helps exclude local and generated files from version control.
• Automated tests help verify behavior after code or dependency changes.
• A clean-environment installation test helps validate project setup instructions.

28. Conclusion

Structuring Python projects and managing application dependencies are foundational practices for professional software development. A clear directory layout improves maintainability, while virtual environments and dependency declarations help make development and deployment more predictable.

By learning to organize modules and packages, use pip and venv, maintain requirements.txt or pyproject.toml, and test dependency changes, you can build Python applications that are easier to share, maintain, and scale.

Continue learning with the Python Training Course, register through the JustAcademy Course Demo page, or visit the JustAcademy website.