What is parallel computing?

Serial Computing:

• A problem is broken into a discrete series of instructions.
• Instructions are executed serially one after another.
• Executed on a single processor.
• Only one instruction may execute at any moment in time.

Parallel Computing:

• A problem is broken into a discrete series of instructions that can be solved concurrently.
• Each part is further broken down to a series of instructions.
• Instructions from each part execute simultaneously on different processors.
• An overall control/coordination mechanism is employed.

Serial and Parallel Computing

Speedup, Efficiency & Amdahl’s Law

• \(T(p, N)\): Time to solve a problem of total size N on p processors.

• Parallel Speedup: Let \(S(p, N) = \frac{T(1, N)}{T(p, N)}\) be the parallel speedup.

• Parallel Efficiency: Let \(E(p, N) = \frac{S(p, N)}{p}\) be the parallel efficiency.

Weak versus strong scaling

Strong scaling: Is defined as how the solution time varies with the number of processors for a fixed total problem size.

Weak scaling: Is defined as how the solution time varies with the number of processors for a fixed problem size per processor.

The Problem: Your Code is Too Slow

Scenario:
You need to price 10,000 exotic options using Monte Carlo simulation.

• Each option requires 100,000 simulation paths
• Each simulation path takes 0.1 seconds
• Total time:
  \(10,\!000 \times 0.1 = 1,\!000\) seconds ≈ 17 minutes

The Reality:
Your laptop has 14 CPU cores, but Python is only using one of them!

The Goal:
Use all cores → Reduce time to ≈ 1.5 minutes

Sequential vs. Parallel Execution

Sequential and Parallel Execution

Why Quantitative Macro Needs Parallelism

• CPU-Intensive Tasks (Model Solving & Estimation)
  • Global Solution Methods: Value Function Iteration (VFI) and projection methods over large state spaces.
  • Bayesian Estimation: Running multiple Markov Chain Monte Carlo (MCMC) chains for parameter estimation.
  • Simulating Method of Moments (SMM): Repeatedly solving models to match empirical moments.
• I/O-Intensive Tasks (Data Pipeline)
  • API Data Aggregation: Fetching thousands of series simultaneously from FRED, World Bank, or IMF APIs.
  • Large Panel Processing: Reading census data or administrative tax records from disk.

• The Two Types of Waiting
  • CPU-bound: Waiting for the model to converge or the grid search to finish → Multiprocessing
  • I/O-bound: Waiting for the web scraper or API to return data → Threading or Async I/O
• Python’s Parallel Toolkit

First Look: Sequential vs. Parallel

Sequential (what you’re used to):

results = []
for item in data:
    result = slow_function(item)
    results.append(result)

Parallel (what we’ll learn):

from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor() as executor:
    results = list(executor.map(slow_function, data))

That’s it! Two extra lines of code can give you 4–8× speedup on multi-core machines.

What is a Thread?

• Process:
  • An independent program, with its own memory space
    
  • Heavyweight to create
    
  • Supports true parallelism
• Thread:
  • Lives inside a process, sharing that process’s memory
    
  • Lightweight to create and manage
    
  • Enables concurrency (but not true parallelism in Python*)

The Global Interpreter Lock (GIL)

What is the GIL?

• A mutex (mutual exclusion lock) implemented in CPython.
• Only one thread can execute Python bytecode at a time.
• Protects memory management within the Python interpreter.

Implications:

• CPU-bound code: Threads do not speed up execution (because of GIL).
• I/O-bound code: Threads can help significantly.

Why Do Threads Help with I/O-Bound Code?

• The GIL is released during I/O operations (like reading files, network calls, waiting for input/output).
• While one thread waits for data (e.g., disk or network), another thread can use the CPU to continue processing Python code.

# Timeline illustration:
# Time →
# T1:  |---Python code---|.......I/O wait.......|
# T2:                    |---Python code---|...I/O wait...|

I/O-Bound vs. CPU-Bound

ThreadPoolExecutor: The Simple Way

from concurrent.futures import ThreadPoolExecutor
import time

def fetch_stock_data(ticker):
    """Simulate fetching data (I/O operation)"""
    time.sleep(0.5)  # Simulate network delay
    return {"ticker": ticker, "price": 100.0}

tickers = ["AAPL", "GOOGL", "MSFT", "AMZN"]

%%time
# Parallel fetching
with ThreadPoolExecutor() as executor:
    results = list(executor.map(fetch_stock_data, tickers))

CPU times: user 440 μs, sys: 465 μs, total: 905 μs
Wall time: 505 ms

%%time
# Sequential fetching
results = [fetch_stock_data(ticker) for ticker in tickers]

CPU times: user 230 μs, sys: 178 μs, total: 408 μs
Wall time: 2.01 s

Pattern 1: executor.map()

Use when:

• You have the same function to run
• Many inputs
• The order of results matters

def process(item):
    return item * 2


items = [1, 2, 3, 4, 5]
with ThreadPoolExecutor() as executor:
    results = list(executor.map(process, items))
print(results)

[2, 4, 6, 8, 10]

Pattern 2: executor.submit() + as_completed()

Use when:

• You want results as soon as they finish (not in input order)
• You want progress feedback or early termination
• Tasks have heterogeneous/variable completion times

from concurrent.futures import as_completed

tickers = ["AAPL", "GOOGL", "MSFT", "AMZN"]

with ThreadPoolExecutor() as executor:
    # Submit tasks
    futures = {executor.submit(fetch_stock_data, t): t for t in tickers}
    # Process results as they complete
    for future in as_completed(futures):
        ticker = futures[future]
        result = future.result()
        print(f"{ticker}: got data!")

AAPL: got data!
AMZN: got data!
GOOGL: got data!
MSFT: got data!

Handling Exceptions in Threads

def risky_fetch(ticker):
    if ticker == "BAD":
        raise ValueError(f"Invalid ticker: {ticker}")
    return {"ticker": ticker, "price": 100.0}


tickers = ["AAPL", "BAD", "MSFT"]

with ThreadPoolExecutor() as executor:
    futures = {executor.submit(risky_fetch, t): t for t in tickers}
    for future in as_completed(futures):
        ticker = futures[future]
        try:
            result = future.result()
            print(f"{ticker}: {result}")
        except Exception as e:
            print(f"{ticker}: ERROR - {e}")

AAPL: {'ticker': 'AAPL', 'price': 100.0}
BAD: ERROR - Invalid ticker: BAD
MSFT: {'ticker': 'MSFT', 'price': 100.0}

Threading: Key Takeaways

When to Use Threading

• Fetching data from multiple sources
• Reading/writing multiple files
• Any task where you’re waiting for external resources

When NOT to Use Threading

• Heavy computations (e.g., Monte Carlo, optimization)
• Number crunching tasks
  → Use multiprocessing instead!

Best Practices

• Use ThreadPoolExecutor (not raw threads)
• Always handle exceptions
• Use context managers (with statement)

Why Multiprocessing?

The GIL Problem:

• Threads share one GIL (Global Interpreter Lock)
  • Only one thread runs Python code at a time.
  • CPU-bound code does not speed up with threading.

The Solution: Multiprocessing

• Use separate processes (separate memory, separate GIL).
• The Challenge: Native Python multiprocessing is notoriously fragile in Jupyter Notebooks (serialization errors!).

The Tool: joblib

• The standard for scientific computing (used by scikit-learn).
• Notebook-friendly: Works where standard libraries fail.
• Easy Syntax: Resembles a simple loop.

The Syntax: From Loop to Parallel

Sequential Loop (List Comprehension):

[slow_function(x) for x in data]

Parallel Loop (Joblib):

from joblib import Parallel, delayed

# n_jobs=-1 means "use all CPU cores"
Parallel(n_jobs=-1)(delayed(slow_function)(x) for x in data)

Example: Parallel Monte Carlo

Let’s estimate using all your laptop’s cores.

from joblib import Parallel, delayed
import numpy as np


def monte_carlo_pi(n_samples):
    """Simulate points to estimate Pi"""
    x = np.random.random(n_samples)
    y = np.random.random(n_samples)
    inside = np.sum(x**2 + y**2 <= 1)
    return 4 * inside / n_samples

Run in Parallel:

%%time

n_tasks = 100
samples_per_task = 1_000_000
results = Parallel(n_jobs=-1)(
    delayed(monte_carlo_pi)(samples_per_task) for _ in range(n_tasks)
)

pi_estimate = np.mean(results)
print(f"Estimated pi: {pi_estimate:.5f}")

Estimated pi: 3.14160
CPU times: user 68.7 ms, sys: 62.6 ms, total: 131 ms
Wall time: 879 ms

When to use what?

Debugging Parallel Code

Parallel bugs can be hard to find because of:

• Non-deterministic behavior
• Errors in worker processes
• Difficulty reproducing issues

Strategies for Debugging

1. Start sequential: Make sure your code works with max_workers=1.
2. Catch exceptions: Always wrap future.result() (or similar) in try/except.
3. Use logging: Print statements can get mixed up—prefer proper logging.

What is Numba?

Python is dynamic and flexible, but this comes at a cost: loops are slow. Every iteration requires type-checking and memory allocation.

Numba is a Just-In-Time (JIT) compiler that translates a subset of Python and NumPy code into fast machine code (using LLVM).

• @njit (nopython mode): The core decorator. It attempts to compile the decorated function entirely without the Python interpreter. If it fails, it raises an error.
• parallel=True: Enables automatic parallelization of array operations and explicit parallel loops.
• prange: “Parallel Range”. A replacement for range that tells Numba: “It is safe to run iterations of this loop in any order, on different CPU cores.”

Scenario: Multi-Sector Aggregate Risk

Imagine an economy with 3 major sectors (e.g., Manufacturing, Services, Tech). We want to estimate the “Tail Risk” (5th percentile outcome) of Aggregate GDP growth.

• We simulate 1 million stochastic scenarios.
• Each sector has its own productivity trend (\(\mu_i\)) and volatility (\(\sigma_i\)).

import numpy as np
import time

def simulate_gdp_tail_risk(n_sims, weights, mu, sigma):
    """
    Simulates aggregate GDP growth scenarios to find the 5th percentile (Tail Risk).
    """
    gdp_growth = np.empty(n_sims)

    # --- The Bottleneck ---
    # Python loops have high overhead for simple math
    for i in range(n_sims):
        sector_shocks = np.random.randn(len(weights))
        sector_growth = sector_shocks * sigma + mu
        gdp_growth[i] = np.dot(weights, sector_growth)
    # ----------------------

    gdp_growth.sort()
    # Return the 5th percentile (Left-tail risk)
    return gdp_growth[int(n_sims * 0.05)]

We now run the simulation.

# Parameters
n_sims = int(1e6)
sector_weights = np.array([0.2, 0.5, 0.3])  # Manuf, Services, Tech
mu = np.array([0.01, 0.02, 0.03])  # Trend growth
sigma = np.array([0.05, 0.04, 0.08])  # Volatility

# Timing
start_time = time.time()
risk_result = simulate_gdp_tail_risk(n_sims, sector_weights, mu, sigma)
pure_time = time.time() - start_time

print(f"5% Tail Risk (GDP Growth): {risk_result:.4f}")
print(f"Python Time: {pure_time:.4f} seconds")

5% Tail Risk (GDP Growth): -0.0329
Python Time: 1.0558 seconds

The Numba Solution (@njit)

We add the @njit decorator. This compiles the loop into machine code, eliminating Python’s interpreter overhead.

import numba

@numba.njit
def simulate_gdp_numba(n_sims, weights, mu, sigma):
    gdp_growth = np.empty(n_sims)
    for i in range(n_sims):
        sector_shocks = np.random.randn(len(weights))
        sector_growth = sector_shocks * sigma + mu
        gdp_growth[i] = np.dot(weights, sector_growth)

    gdp_growth.sort()
    return gdp_growth[int(n_sims * 0.05)]

# Warmup (Compile the function)
_ = simulate_gdp_numba(100, sector_weights, mu, sigma)

# Timing
start_time = time.time()
risk_numba = simulate_gdp_numba(n_sims, sector_weights, mu, sigma)
numba_time = time.time() - start_time

print(f"Numba Time: {numba_time:.4f} seconds")
print(f"Speedup: {pure_time / numba_time:.1f}x")

Numba Time: 0.1667 seconds
Speedup: 6.3x

The Numba Parallel Solution (prange)

Now we use all CPU cores.

1. Add parallel=True to the decorator.
2. Replace range with numba.prange.

@numba.njit(parallel=True)
def simulate_gdp_parallel(n_sims, weights, mu, sigma):
    gdp_growth = np.empty(n_sims)

    # prange tells Numba: "You can split this loop across cores"
    for i in numba.prange(n_sims):
        sector_shocks = np.random.randn(len(weights))
        sector_growth = sector_shocks * sigma + mu
        gdp_growth[i] = np.dot(weights, sector_growth)
    gdp_growth.sort()
    return gdp_growth[int(n_sims * 0.05)]

# Warmup
_ = simulate_gdp_parallel(100, sector_weights, mu, sigma)
start_time = time.time()
risk_parallel = simulate_gdp_parallel(n_sims, sector_weights, mu, sigma)
parallel_time = time.time() - start_time

print(f"Parallel Time: {parallel_time:.4f} seconds")
print(f"Total Speedup vs Python: {pure_time / parallel_time:.1f}x")

Parallel Time: 0.0749 seconds
Total Speedup vs Python: 14.1x