File I/O and Exception Handling in Python

Scenario context

Python file operations should always use the with statement to ensure resource cleanup. Use the EAFP (Easier to Ask Forgiveness than Permission) style with specific try/except/else/finally blocks to write clean, race-free error handling, and specify encoding='utf-8' explicitly for all text files.

Picture this: it's 5 PM on a Friday. A developer pushes a new service to production. The deployment succeeds, but…

This walkthrough focuses on The Config File That Took Down a Friday Deployment.

Before making a design choice, clarify:

• Picture this: it's 5 PM on a Friday.
• A developer pushes a new service to production.
• The deployment succeeds, but five minutes later the service is dead — a cascade of AttributeError: 'NoneType' object has no attribute 'get' errors flooding the logs.

Concepts in play: python-concurrency-nogil, database-indexing, stack-and-queue-data-structures, api-gateway

Stage 1 — Opening Files the Pythonic Way: open(), Modes, and the with Statement

• The open() built-in is the gateway to all file operations in Python.
• It takes a file path and an optional mode string that controls how the file is opened.
• File Opening Modes The mode "r" is the default — calling open("myfile.txt") is identical to open("myfile.txt", "r") .
• Reading a File: Three Methods Once a file is open, you have three ways to read its contents: For large files, the most memory-efficient approach is to iterate the file object directly — Python reads one line at a time w…

# All three reading methods demonstrated

with open("sample.txt", "r", encoding="utf-8") as f:
    full_text = f.read()        # Reads entire file into one string
    print(full_text)

with open("sample.txt", "r", encoding="utf-8") as f:
    first_line = f.readline()   # Reads exactly one line (including \n)
    second_line = f.readline()  # Reads the next line
    print(first_line, second_line)

with open("sample.txt", "r", encoding="utf-8") as f:
    all_lines = f.readlines()   # Returns a list of all lines
    for line in all_lines:
        print(line.strip())     # .strip() removes the trailing \n

# Memory-efficient line-by-line reading — ideal for large log files
with open("large_logfile.txt", "r", encoding="utf-8") as f:
    for line in f:              # File object is iterable
        process(line.strip())   # Never loads whole file into memory

Observable outcome: Reading a File: Three Methods Once a file is open, you have three ways to read its contents: For large files, the most memory-efficient approach is to iterate the file object directly — Python reads one line at a time w…

Stage 2 — Python's Exception System: Hierarchy, try/except/else/finally, and Advanced Patterns

• Before you can handle exceptions intelligently, you need to understand where they come from and how they relate to each other.
• The Exception Hierarchy You Actually Need to Know Python exceptions form a class hierarchy.
• Every exception is an object that inherits from BaseException .
• The classes you'll interact with most are: This hierarchy matters for your except clauses.

BaseException
├── SystemExit              ← raised by sys.exit() — do not catch unless cleaning up
├── KeyboardInterrupt       ← Ctrl+C — do not swallow silently
└── Exception               ← Parent of all "normal" program errors
    ├── ValueError          ← Bad argument value (e.g., int("abc"))
    ├── TypeError           ← Wrong type (e.g., "2" + 2)
    ├── AttributeError      ← Object has no attribute
    ├── IndexError          ← List index out of range
    ├── KeyError            ← Dict key not found
    ├── StopIteration       ← Iterator exhausted — used by for loops
    ├── RuntimeError        ← Miscellaneous runtime errors
    └── OSError             ← OS-level errors (file, network, permissions)
        ├── FileNotFoundError    ← File or directory does not exist
        ├── PermissionError      ← Insufficient permissions
        ├── IsADirectoryError    ← Tried to open a directory as a file
        └── TimeoutError         ← OS timeout

import json

def parse_config(filepath):
    try:
        # The try block: code that might raise an exception
        with open(filepath, "r", encoding="utf-8") as f:
            data = json.load(f)

    except FileNotFoundError:
        # except: runs only if the named exception was raised
        print(f"Config file '{filepath}' not found.")
        return {}

    except json.JSONDecodeError as e:
        # You can catch multiple exception types, each in its own clause
        print(f"Config file has invalid JSON at line {e.lineno}: {e.msg}")
        return {}

    except (PermissionError, IsADirectoryError) as e:
        # Catch multiple exception types in one clause using a tuple
        print(f"Cannot read config file: {e}")
        return {}

    else:
        # else: runs ONLY if no exception was raised in the try block
        # This is the "success path" — a clean way to signal normal flow
        print(f"Config loaded successfully: {len(data)} keys")
        return data

    finally:
        # finally: runs ALWAYS — whether an exception occurred or not
        # Use it for cleanup that must happen regardless of outcome
        print("parse_config() finished.")

Observable outcome: The classes you'll interact with most are: This hierarchy matters for your except clauses.

Stage 3 — How Python Finds the Right Handler: Exception Propagation Through the Call Stack

• Understanding what happens after an exception is raised requires a mental model of the call stack and how Python searches for a matching handler.
• The diagram below traces a FileNotFoundError raised inside open() as it travels up a three-function call stack.
• Python searches for a matching except clause in the current frame first, then in each caller frame going upward.
• The finally block in every frame that has one will execute during unwinding, regardless of whether a handler is found.

graph TD
    A[main calls load_config] --> B[load_config calls open]
    B --> C{File exists?}
    C -->|Yes| D[Return file handle]
    C -->|No| E[FileNotFoundError raised]
    E --> F{except clause in open frame?}
    F -->|No| G[Unwind to load_config frame]
    G --> H{except FileNotFoundError in load_config?}
    H -->|Yes| I[Handler executes - exception resolved]
    H -->|No| J[Unwind to main frame]
    J --> K{except clause in main?}
    K -->|Yes| L[Handler executes - exception resolved]
    K -->|No| M[Interpreter catches - full traceback printed]
    I --> N[finally blocks run on the way out]
    L --> N
    D --> O[else block runs - success path]

Observable outcome: The finally block in every frame that has one will execute during unwinding, regardless of whether a handler is found.

Stage 4 — File I/O in the Real World: CSV, JSON Configs, Log Files, and Binary Data

• The open-read-close pattern extends to several common file formats Python developers encounter daily.
• Processing CSV Files with the csv Module Never parse CSV by splitting on commas manually — quoted fields with embedded commas will break it silently.
• Python's csv module handles all edge cases: Loading a JSON Configuration File JSON configuration loading is one of the most common file I/O tasks in Python services: Note the use of pathlib.Path — it is the modern, obje…
• Appending to a Log File Append mode ( "a" ) is perfect for log files — it adds to the end without erasing existing content, and it creates the file if it does not yet exist: Reading Binary Files (Images, PDFs) Binary mo…

import csv

def read_employees(filepath):
    employees = []
    try:
        with open(filepath, "r", encoding="utf-8", newline="") as f:
            # newline="" is required for csv.reader to handle line endings correctly
            reader = csv.DictReader(f)
            for row in reader:
                # Each row is an OrderedDict keyed by the header row
                employees.append({
                    "name": row["name"],
                    "department": row["department"],
                    "salary": float(row["salary"]),
                })
    except FileNotFoundError:
        print(f"CSV file not found: {filepath}")
    except KeyError as e:
        print(f"Missing expected column: {e}")
    except ValueError as e:
        print(f"Could not parse salary as a number: {e}")
    return employees

import json
from pathlib import Path

DEFAULT_CONFIG = {
    "log_level": "INFO",
    "database": {"host": "localhost", "port": 5432},
    "max_connections": 10,
}

def load_config(config_path="config.json"):
    path = Path(config_path)
    try:
        with path.open("r", encoding="utf-8") as f:
            user_config = json.load(f)
        # Merge user config on top of defaults — user values win
        return {**DEFAULT_CONFIG, **user_config}
    except FileNotFoundError:
        print(f"No config found at {config_path!r} — using defaults.")
        return DEFAULT_CONFIG.copy()
    except json.JSONDecodeError as e:
        print(f"Invalid JSON in config: {e}")
        return DEFAULT_CONFIG.copy()

Observable outcome: Appending to a Log File Append mode ( "a" ) is perfect for log files — it adds to the end without erasing existing content, and it creates the file if it does not yet exist: Reading Binary Files (Images, PDFs) Binary mo…

Stage 5 — Three Worked Examples: From Fragile to Production-Ready

• This section walks through three complete, runnable examples that apply everything covered above.
• Each one starts with context, shows the full implementation, then highlights what makes it robust.
• Example 1: Robust Config Loader with Defaults and Validation A production config loader needs to handle missing files, malformed JSON, and missing required keys — all without crashing the calling service: This example u…
• Example 2: Streaming Log Processor with Error Recovery A streaming processor reads a potentially large log file line by line.

import json
from pathlib import Path

REQUIRED_KEYS = {"database_url", "secret_key", "debug"}

def load_application_config(path="config.json"):
    """
    Load application config from a JSON file.
    Returns a config dict on success.
    Raises RuntimeError with a clear message on unrecoverable failure.
    """
    config_path = Path(path)

    try:
        with config_path.open("r", encoding="utf-8") as f:
            config = json.load(f)
    except FileNotFoundError:
        raise RuntimeError(
            f"Required config file not found: {config_path.resolve()}\n"
            "Create config.json before starting the service."
        ) from None
    except json.JSONDecodeError as e:
        raise RuntimeError(
            f"Config file contains invalid JSON: {e}\n"
            f"Fix line {e.lineno} in {config_path}."
        ) from e

    missing = REQUIRED_KEYS - set(config.keys())
    if missing:
        raise RuntimeError(
            f"Config is missing required keys: {sorted(missing)}"
        )

    return config

# Usage
try:
    config = load_application_config()
    print(f"Service starting with debug={config['debug']}")
except RuntimeError as e:
    print(f"Startup failed: {e}")
    raise SystemExit(1)

import re
from datetime import datetime

LOG_PATTERN = re.compile(
    r"\[(?P<timestamp>[^\]]+)\]\s+(?P<level>INFO|WARN|ERROR)\s+(?P<message>.+)"
)

def process_log_file(filepath, target_level="ERROR"):
    """
    Stream-parse a log file and yield entries at or above target_level.
    Skips malformed lines and reports a summary at the end.
    """
    counts = {"parsed": 0, "skipped": 0, "matched": 0}
    results = []

    try:
        with open(filepath, "r", encoding="utf-8", errors="replace") as f:
            for line_number, line in enumerate(f, start=1):
                line = line.strip()
                if not line:
                    continue

                match = LOG_PATTERN.match(line)
                if not match:
                    counts["skipped"] += 1
                    continue

                counts["parsed"] += 1
                if match.group("level") == target_level:
                    counts["matched"] += 1
                    results.append({
                        "line": line_number,
                        "timestamp": match.group("timestamp"),
                        "message": match.group("message"),
                    })

    except FileNotFoundError:
        print(f"Log file not found: {filepath}")
        return [], counts
    except OSError as e:
        print(f"Could not read log file: {e}")
        return [], counts
    else:
        print(
            f"Processed {counts['parsed']} lines, "
            f"skipped {counts['skipped']} malformed, "
            f"found {counts['matched']} {target_level} entries."
        )

    return results, counts

# Usage
errors, stats = process_log_file("service.log", target_level="ERROR")
for entry in errors:
    print(f"Line {entry['line']}: {entry['message']}")

Observable outcome: Example 2: Streaming Log Processor with Error Recovery A streaming processor reads a potentially large log file line by line.

Stage 6 — Deep Dive: How Python's Exception System Propagates Errors Across the Call Stack

• When an exception is raised, Python constructs an exception object (a regular Python object inheriting from BaseException ) and begins unwinding the call stack.
• At each frame, Python checks whether the current try block has a matching except clause.
• If it does, execution jumps to that handler.
• If not, the frame is popped and unwinding continues up the stack.

Observable outcome: If not, the frame is popped and unwinding continues up the stack.

End-to-end outcome

After working through the stages, you should be able to explain:

• Every time you write f = open(...) without a with block, you are betting that no exception will occur between that line and f.close() .
• You will eventually lose that bet.

Alternatives considered

Decision Guide: Choosing the Right File and Exception Strategy

Decision Guide: Choosing the Right File and Exception Strategy

Failure and degraded modes

Trade-offs & Failure Modes: File I/O and Exception Handling Pitfalls

• Detection: The most common production failure is reading large files fully into memory with f.read() .
• User or operator impact: The most common production failure is reading large files fully into memory with f.read() .