【中文说明】用于计算餐饮菜单的单份成本、食材成本率、贡献毛利和建议售价，并结合销量进行菜单分类。提供供应商价格、采购单位、可用率、配方份量、税率和销售数据，要求输出计算依据及调价、分量或菜单优化建议。 For cafes, restaurants, bakeries, and food trucks: turns supplier prices, yields, and recipes into exact cost per portion, food cost percent on the tax-free price, contribution margin, and a suggested price, then applies menu engineering (Star, Plowhorse, Puzzle, Dog) with a tested Python calculator.
---
name: menu-food-cost-calculator
description: Costs recipes and menu items for cafes, restaurants, bakeries, food trucks, and caterers - converts purchase prices and yields into an exact cost per portion, food cost percentage on the tax-free price, contribution margin, and a suggested price at a target food cost, then classifies items with menu engineering (Star, Plowhorse, Puzzle, Dog) and recommends price, portion, and menu changes. Use when a user asks "what does this dish cost me?", "how should I price my menu?", "why is my food cost so high?", or shares recipes with supplier prices.
---
# Menu Food Cost and Pricing Calculator
You help small food businesses know what every plate really costs and price it with confidence. You work from real purchase prices and recipes, you show the math, and you think about margin in money, not only in percentages.
## Files in this skill
- `scripts/cost_menu.py` - costs every recipe from a JSON costing sheet, suggests prices, and runs menu engineering (Python 3 standard library only)
- `references/food-cost-basics.md` - yield, as-purchased versus edible cost, food cost percent, taxes, and common costing mistakes
- `references/pricing-strategies.md` - target-percent pricing, margin-based pricing, rounding, and menu engineering actions
- `templates/recipe-costing-sheet.md` - the JSON costing sheet the script reads, plus the report layout
- `examples/example-cafe-menu.md` - a worked review of a five-item cafe menu
## Workflow
### 1. Collect the inputs
Ask for or confirm:
- Currency, and whether menu prices include VAT or sales tax (and the rate).
- Target food cost percent (typical ranges are in `references/food-cost-basics.md`; default 30).
- For each ingredient: purchase price, pack size and unit, and yield (usable share after trimming, peeling, cooking loss, or spoilage).
- For each item: recipe quantities as prepared amounts, number of portions per batch, current menu price, packaging or garnish per portion, and weekly sales if known.
If something is missing, use a clearly labeled assumption (for example "yield 90 percent assumed for avocados") and list it in the report.
### 2. Build the costing sheet
Fill in `templates/recipe-costing-sheet.md` as JSON. Use units the script knows (g, kg, ml, l, oz, lb, each). If an ingredient is bought by the piece but used by weight, weigh one piece and convert; never mix dimensions.
### 3. Run the calculator
```bash
python3 scripts/cost_menu.py menu.json
python3 scripts/cost_menu.py menu.json --target 28
python3 scripts/cost_menu.py menu.json --json
```
The table shows cost per portion, menu price, net price without tax, food cost percent, contribution margin (net price minus cost), the price at the target food cost (rounded up), and the menu engineering class when weekly sales are given for every item. Errors (unknown ingredients, unit mismatches) and HIGH findings make the exit code 1.
If you cannot run the script, do the same calculation by hand, line by line, and say so.
### 4. Recommend
Use `references/pricing-strategies.md`:
1. Fix data errors first and rerun.
2. For HIGH and WARN items choose between raising the price, trimming the portion, changing an expensive ingredient, or accepting a higher percent because the money margin is strong. Name the trade-off.
3. Use the menu engineering class to decide where an item belongs on the menu and whether to promote, reprice, rework, or remove it.
4. Rerun with the proposed changes to show the before and after.
### 5. Report
Use the report layout in `templates/recipe-costing-sheet.md`, as in `examples/example-cafe-menu.md`.
## Rules
- Show the formula for at least one item so the owner can check it: cost per portion / net price x 100.
- Never treat the price at target as an instruction to lower an existing price; it is a benchmark.
- Do not give tax or legal advice; only apply the tax rate the user provides.
- Respect allergens and dietary claims when suggesting substitutions, and never suggest lowering food safety or quality standards.
- Recheck costs when supplier prices change by more than about 5 percent.
FILE:references/food-cost-basics.md
# Food cost basics
## Key terms
- **As-purchased (AP) cost**: what you pay for the pack, case, or piece.
- **Yield percent**: the usable share after trimming, peeling, deboning, cooking loss, or spoilage. Salmon fillet trimmed of skin and pin bones might yield 85 percent; whole avocados where 1 in 10 is unusable yield 90 percent when counted by the piece.
- **Edible portion (EP) cost** = AP cost per unit / (yield percent / 100). Recipes list prepared, usable quantities, so they are costed at EP cost.
- **Plate cost (cost per portion)** = sum of ingredient EP costs for the batch / portions + extras per portion (packaging, napkin, garnish, sauce cup).
- **Net price** = menu price / (1 + tax rate), when menu prices include VAT or sales tax. Food cost must be measured against the money you keep, not the tax you collect.
- **Food cost percent** = plate cost / net price x 100.
- **Contribution margin** = net price - plate cost. This is the money each sale leaves to pay labor, rent, and profit.
## Worked formula
Salmon fillet bought at 32.00 per kg with 85 percent yield:
- AP cost per g = 32.00 / 1000 = 0.032
- EP cost per g = 0.032 / 0.85 = 0.0376
- 160 g portion = 160 x 0.0376 = 6.02
## Typical food cost targets (rough guide)
| Concept | Typical food cost percent |
| --- | --- |
| Coffee and espresso drinks | 15 to 25 |
| Bakery items | 20 to 30 |
| Cafe brunch dishes | 28 to 35 |
| Casual restaurant mains | 28 to 35 |
| Steak and seafood mains | 35 to 45 |
| Pizza | 20 to 28 |
| Catering trays | 25 to 35 |
Your right target depends on labor, rent, and volume. A low-labor item can run a higher food cost percent and still be very profitable.
## Common costing mistakes
1. Forgetting small items: oil, butter for the pan, salt, garnish, sauces, and takeaway packaging. Add them or use extras_per_portion.
2. Using AP cost without yield for proteins and produce.
3. Measuring food cost against prices that include tax.
4. Costing the recipe card instead of what the kitchen actually plates (portion creep). Weigh five real portions.
5. Old supplier prices. Update the sheet when a price moves by about 5 percent or more.
6. Mixing units: an ingredient bought by the piece but used by weight needs one piece weighed.
7. Ignoring waste and staff meals; track them separately and compare actual food cost (from inventory) with this theoretical cost.
## Theoretical versus actual food cost
This skill calculates theoretical cost: what food should cost if recipes are followed. Actual food cost = (opening inventory + purchases - closing inventory) / net food sales. A gap of more than about 2 to 3 points usually means waste, portion creep, theft, or wrong prices on the sheet.
FILE:references/pricing-strategies.md
# Pricing strategies and menu engineering
## Ways to set a price
1. **Target food cost percent**: price = plate cost / target x (1 + tax rate), rounded up. Simple, and the script's "AT TARGET" column. Weak spot: cheap items end up underpriced and expensive proteins overpriced.
2. **Contribution margin**: decide the money each item must earn (for example at least 5.00 for a main), then price = (plate cost + margin) x (1 + tax rate). Better for high-cost proteins.
3. **Market check**: compare with three to five similar places nearby. Price perception matters as much as cost.
4. **Blend**: start from the target price, check the margin in money, then sanity-check against the market.
## Rounding and presentation
- Round up to the step your menu uses (0.10, 0.50, or whole numbers). Upscale menus often use whole numbers without currency signs; casual menus often end in .50 or .90.
- Avoid many small increases across the whole menu at once; raise the items with the weakest margin first.
- Keep price gaps logical: an oat milk upgrade should cover its extra cost (oat drink often costs about twice as much as dairy milk per liter).
## Menu engineering
Needs weekly sales for every item. The script uses:
- **Popularity line**: an item is popular if it sells at least 70 percent of an equal share (with 5 items, 0.7 x 20 percent = 14 percent of units sold).
- **Margin line**: the sales-weighted average contribution margin.
| Class | Popularity | Margin | What to do |
| --- | --- | --- | --- |
| Star | high | high | Keep quality and portion consistent, place it in the best menu spot, small price increases are usually safe. |
| Plowhorse | high | low | Raise price a little, trim cost (portion, garnish, supplier), or pair it with a high-margin add-on. Do not remove it. |
| Puzzle | low | high | Promote it: better menu placement, a description, staff recommendation, a photo. Check the price is not scaring guests. |
| Dog | low | low | Rework the recipe or price, or remove it, unless it serves a purpose (kids menu, dietary option, signature item). |
## Choosing a fix for a high food cost item
| Option | Good when | Risk |
| --- | --- | --- |
| Raise the price | the item is popular and the market allows it | fewer sales if the jump is large |
| Trim the portion | portions are larger than guests expect | guests notice; keep value perception |
| Swap an ingredient | a cheaper equal-quality option exists | allergen and taste changes; update the menu text |
| Accept a higher percent | the money margin is the highest on the menu | needs volume to pay off |
| Remove the item | it is a Dog with no strategic role | regulars may miss it |
Always rerun the calculator with the proposed change and show before and after.
FILE:templates/recipe-costing-sheet.md
# Recipe costing sheet (input for scripts/cost_menu.py)
Save as `menu.json`. Quantities in recipes are prepared (usable) amounts.
```json
{
"currency": "EUR",
"target_food_cost_pct": 30,
"menu_price_includes_tax_pct": 10,
"price_rounding": 0.10,
"ingredients": [
{"name": "flour", "price": 0.95, "per": "1 kg"},
{"name": "butter", "price": 9.80, "per": "1 kg"},
{"name": "eggs", "price": 3.60, "per": "12 each"},
{"name": "blueberries", "price": 16.00, "per": "1 kg", "yield_pct": 95}
],
"recipes": [
{
"name": "Blueberry Muffin",
"portions": 12,
"menu_price": 3.20,
"sold_per_week": 90,
"extras_per_portion": 0.06,
"items": [["flour", "500 g"], ["butter", "180 g"], ["eggs", "3 each"], ["blueberries", "300 g"]]
}
]
}
```
Field notes:
- `per`: the pack you buy, as "<amount> <unit>" (g, kg, mg, ml, cl, dl, l, oz, lb, each).
- `yield_pct`: 1 to 100, default 100.
- `menu_price_includes_tax_pct`: 0 if menu prices are shown without tax.
- `sold_per_week`: give it for every item (or none) to get menu engineering classes.
- `extras_per_portion`: packaging, napkins, garnish, sauce cups, in money.
Run: `python3 scripts/cost_menu.py menu.json [--target 30] [--json]`
---
# Menu costing report: <business> - <date>
**Target food cost:** <x>% **Prices include tax:** <rate or no> **Currency:** <code>
**Assumptions:** <yields, missing prices, portion weights>
## Results (before)
| Item | Cost/portion | Price | Net | Food % | Margin | At target | Class |
| --- | --- | --- | --- | --- | --- | --- | --- |
## Formula check
<one item worked out line by line>
## What needs attention
1. **<item>** - <finding>. Options: <price / portion / ingredient / accept>. Recommendation: <one>.
## Proposed changes and results (after)
<changes, then the new table or the changed rows>
## Menu engineering actions
- Stars: <items and action>
- Plowhorses: <items and action>
- Puzzles: <items and action>
- Dogs: <items and action>
## Next steps
- <weigh real portions, update supplier prices, track actual food cost monthly>
FILE:examples/example-cafe-menu.md
# Example: a five-item cafe menu
**User:** "We are a small brunch cafe. Prices include 10 percent VAT and I want about 30 percent food cost. Here are my supplier prices and recipes. Why is my margin so thin?"
The sheet has 16 ingredients and 5 items with weekly sales (avocado yield 90 percent because about 1 in 10 is unusable; salmon 85 percent after trimming).
**Command:**
```bash
python3 scripts/cost_menu.py cafe-menu.json
```
**Output (before):**
```
ITEM COST PRICE NET FOOD% MARGIN AT TARGET CLASS
Avocado Toast 3.13 9.50 8.64 36.2 5.51 11.50 Star
Salmon Spinach Bowl 8.02 14.50 13.18 60.8 5.16 29.50 Puzzle
Flat White 0.72 3.80 3.45 21.0 2.73 2.70 Plowhorse
Oat Flat White 0.91 4.20 3.82 23.9 2.91 3.40 Plowhorse
Blueberry Muffin 0.79 3.20 2.91 27.2 2.12 3.00 Dog
Findings (8):
[HIGH] Salmon Spinach Bowl: food cost 60.8% is far above the 30% target; price EUR 29.50 or cut cost 4.06 per portion
[WARN] Avocado Toast: food cost 36.2% is above the 30% target; price at target would be EUR 11.50
[INFO] Salmon Spinach Bowl: salmon fillet is 76% of the cost; its price or portion matters most
[INFO] menu engineering: weighted average margin EUR 3.22, popularity line 14.0% of items sold
[INFO] ingredient 'truffle oil' is not used in any recipe
```
---
# Menu costing report: brunch cafe - October
**Target food cost:** 30% **Prices include tax:** 10% VAT **Currency:** EUR
**Assumptions:** avocado yield 90%, salmon 85%, spinach 90%; extras 0.10 per dish, 0.12 per coffee (cup and lid), 0.06 per muffin.
## Formula check (Salmon Spinach Bowl, before)
- Salmon 160 g x (32.00 / 1000 / 0.85) = 6.02
- Spinach 70 g x (14.00 / 1000 / 0.90) = 1.09; egg 0.30; tomatoes 0.36; olive oil 0.15; extras 0.10
- Cost per portion = 8.02; net price = 14.50 / 1.10 = 13.18; food cost = 8.02 / 13.18 x 100 = 60.8%
## What needs attention
1. **Salmon Spinach Bowl (HIGH, Puzzle)** - 60.8% food cost; salmon is 76% of the cost. Pricing it at target (29.50) is unrealistic for a cafe. Recommendation: reduce salmon to 120 g (still a generous portion for a bowl) and raise the price to 17.50; accept about 40% food cost because the margin becomes the highest on the menu.
2. **Avocado Toast (WARN, Star)** - 36.2%. It is the best-selling dish, so a 1.00 increase to 10.50 is low risk.
3. **Flat White and Oat Flat White (Plowhorses)** - healthy percentages (21 to 24%) but small margins; do not discount. Keep the oat surcharge at 0.40: the oat drink costs 0.19 more per cup than milk.
4. **Blueberry Muffin (Dog)** - fine percentage, low margin and low sales. Try a bundle with coffee before removing it.
5. **Truffle oil** is on the sheet but in no recipe: remove it from orders or the sheet.
## Proposed changes and results (after)
Avocado Toast 10.50; Salmon Spinach Bowl 120 g salmon at 17.50. Rerun: `python3 scripts/cost_menu.py cafe-menu-revised.json`
```
ITEM COST PRICE NET FOOD% MARGIN AT TARGET CLASS
Avocado Toast 3.13 10.50 9.55 32.8 6.42 11.50 Star
Salmon Spinach Bowl 6.51 17.50 15.91 40.9 9.40 23.90 Puzzle
Flat White 0.72 3.80 3.45 21.0 2.73 2.70 Plowhorse
Oat Flat White 0.91 4.20 3.82 23.9 2.91 3.40 Plowhorse
Blueberry Muffin 0.79 3.20 2.91 27.2 2.12 3.00 Dog
Findings (6):
[WARN] Salmon Spinach Bowl: food cost 40.9% is above the 30% target; price at target would be EUR 23.90
[INFO] menu engineering: weighted average margin EUR 3.56, popularity line 14.0% of items sold
```
Exit code 0. The remaining WARN is accepted on purpose: 9.40 margin per bowl versus 5.16 before.
## Menu engineering actions
- Stars: Avocado Toast - keep the recipe consistent, top of the brunch section.
- Plowhorses: Flat White, Oat Flat White - no discounts; suggest a pastry with every coffee.
- Puzzles: Salmon Spinach Bowl - give it a short description and staff recommendation; check sales after 4 weeks at the new price.
- Dogs: Blueberry Muffin - test a coffee + muffin bundle for 4 weeks, then decide.
## Next steps
- Weigh five real salmon portions this week to confirm the 120 g spec is followed.
- Update supplier prices monthly and rerun the sheet.
- Compare with actual food cost from inventory at month end.
FILE:scripts/cost_menu.py
#!/usr/bin/env python3
"""Cost menu items from recipes and purchase prices, and suggest menu prices.
Usage:
python3 cost_menu.py menu.json [--target 30] [--json]
cat menu.json | python3 cost_menu.py -
Input JSON (see templates/recipe-costing-sheet.md):
{
"currency": "EUR",
"target_food_cost_pct": 30, # optional, default 30 (or --target)
"menu_price_includes_tax_pct": 10, # optional; VAT/sales tax included in menu prices
"price_rounding": 0.10, # optional; suggested prices round UP to this step
"ingredients": [
{"name": "butter", "price": 9.80, "per": "1 kg", "yield_pct": 100}
],
"recipes": [
{"name": "Croissant", "portions": 12, "menu_price": 3.20, "sold_per_week": 180,
"extras_per_portion": 0.05, # optional: packaging, napkin, garnish
"items": [["butter", "600 g"], ["flour", "1 kg"]]}
]
}
Units: g, kg, mg, ml, cl, dl, l, oz, lb, each (also pc, pcs, piece, unit, egg).
Recipe quantities are the prepared (usable) amounts. yield_pct is the usable
share of what you buy after trimming, peeling, cooking loss or spoilage.
Per recipe: cost per portion, food cost percent of the net (tax-free) menu
price, contribution margin, suggested price at the target, and the three
biggest cost drivers. With sold_per_week on every recipe, adds a menu
engineering class (Star, Plowhorse, Puzzle, Dog).
Exit code: 0 ok, 1 errors in the data or HIGH findings, 2 usage or input error.
Standard library only.
"""
import json
import math
import re
import sys
UNITS = { # unit -> (dimension, factor to base unit g / ml / each)
"mg": ("mass", 0.001), "g": ("mass", 1.0), "kg": ("mass", 1000.0),
"oz": ("mass", 28.3495), "lb": ("mass", 453.592),
"ml": ("volume", 1.0), "cl": ("volume", 10.0), "dl": ("volume", 100.0), "l": ("volume", 1000.0),
"each": ("count", 1.0), "pc": ("count", 1.0), "pcs": ("count", 1.0), "piece": ("count", 1.0),
"pieces": ("count", 1.0), "unit": ("count", 1.0), "units": ("count", 1.0), "egg": ("count", 1.0), "eggs": ("count", 1.0),
}
BASE = {"mass": "g", "volume": "ml", "count": "each"}
def usage(msg):
print(f"error: {msg}\n", file=sys.stderr)
print(__doc__.strip().split("\n\n")[1], file=sys.stderr)
sys.exit(2)
def parse_qty(text):
"""'600 g' -> (600.0, 'mass', 600.0 in base units)."""
m = re.fullmatch(r"\s*(\d+(?:[.,]\d+)?)\s*([a-zA-Z]+)?\s*", str(text))
if not m:
raise ValueError(f"cannot read quantity {text!r}")
qty = float(m.group(1).replace(",", "."))
unit = (m.group(2) or "each").lower()
if unit not in UNITS:
raise ValueError(f"unknown unit {unit!r} in {text!r}")
dim, factor = UNITS[unit]
return qty, dim, qty * factor
def round_up(value, step):
return math.ceil(round(value / step, 6)) * step
def analyze(data, target_override=None):
errors, findings = [], []
cur = data.get("currency", "")
target = float(target_override or data.get("target_food_cost_pct", 30))
tax = float(data.get("menu_price_includes_tax_pct", 0))
step = float(data.get("price_rounding", 0.10))
ingredients = {}
for ing in data.get("ingredients", []):
name = str(ing.get("name", "")).strip().lower()
try:
_, dim, base_qty = parse_qty(ing["per"])
price = float(ing["price"])
except (KeyError, ValueError, TypeError) as e:
errors.append(f"ingredient {name or '?'}: {e}")
continue
y = float(ing.get("yield_pct", 100))
if not 0 < y <= 100:
errors.append(f"ingredient {name}: yield_pct must be between 1 and 100")
continue
ingredients[name] = {"dim": dim, "cost_per_base": price / base_qty / (y / 100), "yield": y, "used": False}
results = []
for rec in data.get("recipes", []):
rname = rec.get("name", "?")
portions = float(rec.get("portions", 1) or 1)
lines, bad = [], False
for item in rec.get("items", []):
iname, qtext = str(item[0]).strip().lower(), item[1]
ing = ingredients.get(iname)
if not ing:
errors.append(f"{rname}: unknown ingredient {iname!r} (add it to ingredients)")
bad = True
continue
ing["used"] = True
try:
_, dim, base_qty = parse_qty(qtext)
except ValueError as e:
errors.append(f"{rname}: {e}")
bad = True
continue
if dim != ing["dim"]:
errors.append(f"{rname}: {iname} is bought by {BASE[ing['dim']]} but used by {BASE[dim]} ({qtext}); "
f"convert it (for example weigh one piece)")
bad = True
continue
lines.append((iname, base_qty * ing["cost_per_base"]))
if bad:
continue
batch = sum(c for _, c in lines)
extras = float(rec.get("extras_per_portion", 0))
cost = batch / portions + extras
price = float(rec.get("menu_price", 0))
net = price / (1 + tax / 100) if price else 0.0
pct = 100 * cost / net if net else None
suggested = round_up(cost / (target / 100) * (1 + tax / 100), step)
drivers = sorted(lines, key=lambda x: -x[1])[:3]
r = {"name": rname, "portions": portions, "cost_per_portion": round(cost, 3), "menu_price": price,
"net_price": round(net, 2), "food_cost_pct": None if pct is None else round(pct, 1),
"contribution_margin": round(net - cost, 2) if net else None,
"suggested_price_at_target": round(suggested, 2), "sold_per_week": rec.get("sold_per_week"),
"drivers": [{"ingredient": n, "share_pct": round(100 * c / batch, 1) if batch else 0} for n, c in drivers]}
results.append(r)
if pct is None:
findings.append(("WARN", rname, f"no menu_price; suggested {cur} {suggested:.2f} at {target:g}% food cost"))
elif pct > target + 15:
findings.append(("HIGH", rname, f"food cost {pct:.1f}% is far above the {target:g}% target; "
f"price {cur} {suggested:.2f} or cut cost {cost - net * target / 100:.2f} per portion"))
elif pct > target + 5:
findings.append(("WARN", rname, f"food cost {pct:.1f}% is above the {target:g}% target; "
f"price at target would be {cur} {suggested:.2f}"))
elif pct < target - 15:
findings.append(("INFO", rname, f"food cost only {pct:.1f}%; check the recipe lists every ingredient and portion size"))
if drivers and batch and drivers[0][1] / batch > 0.5:
findings.append(("INFO", rname, f"{drivers[0][0]} is {100 * drivers[0][1] / batch:.0f}% of the cost; "
f"its price or portion matters most"))
sold = [r for r in results if isinstance(r["sold_per_week"], (int, float)) and r["contribution_margin"] is not None]
if sold and len(sold) == len(results) and len(sold) >= 3:
total = sum(r["sold_per_week"] for r in sold)
pop_line = 0.7 / len(sold)
avg_cm = sum(r["contribution_margin"] * r["sold_per_week"] for r in sold) / total if total else 0
for r in sold:
high_pop = total and r["sold_per_week"] / total >= pop_line
high_cm = r["contribution_margin"] >= avg_cm
r["menu_class"] = {(True, True): "Star", (True, False): "Plowhorse",
(False, True): "Puzzle", (False, False): "Dog"}[(bool(high_pop), high_cm)]
findings.append(("INFO", None, f"menu engineering: weighted average margin {cur} {avg_cm:.2f}, "
f"popularity line {100 * pop_line:.1f}% of items sold"))
for name, ing in ingredients.items():
if not ing["used"]:
findings.append(("INFO", None, f"ingredient {name!r} is not used in any recipe"))
return {"currency": cur, "target_pct": target, "tax_pct": tax, "recipes": results,
"errors": errors, "findings": [{"severity": s, "recipe": n, "message": m} for s, n, m in findings]}
def main(argv):
target, as_json, paths = None, False, []
it = iter(argv)
for a in it:
if a == "--target":
try:
target = float(next(it, ""))
except ValueError:
usage("--target needs a number, for example 30")
if not 5 <= target <= 80:
usage("--target should be a food cost percent between 5 and 80")
elif a == "--json":
as_json = True
elif a.startswith("--"):
usage(f"unknown option {a}")
else:
paths.append(a)
if len(paths) != 1:
usage("give exactly one menu JSON file, or - for stdin")
try:
raw = sys.stdin.read() if paths[0] == "-" else open(paths[0], encoding="utf-8").read()
data = json.loads(raw)
except (OSError, ValueError) as e:
print(f"error: cannot read menu JSON: {e}", file=sys.stderr)
return 2
if not data.get("recipes"):
print("error: no recipes in the input", file=sys.stderr)
return 2
rep = analyze(data, target)
if as_json:
print(json.dumps(rep, indent=2))
else:
cur = rep["currency"]
print(f"Target food cost {rep['target_pct']:g}% | prices include {rep['tax_pct']:g}% tax | currency {cur}\n")
print(f"{'ITEM':<22} {'COST':>7} {'PRICE':>7} {'NET':>7} {'FOOD%':>6} {'MARGIN':>7} {'AT TARGET':>9} CLASS")
for r in rep["recipes"]:
pct = "-" if r["food_cost_pct"] is None else f"{r['food_cost_pct']:.1f}"
cm = "-" if r["contribution_margin"] is None else f"{r['contribution_margin']:.2f}"
print(f"{r['name'][:22]:<22} {r['cost_per_portion']:>7.2f} {r['menu_price']:>7.2f} {r['net_price']:>7.2f} "
f"{pct:>6} {cm:>7} {r['suggested_price_at_target']:>9.2f} {r.get('menu_class', '-')}")
print("\nTop cost drivers:")
for r in rep["recipes"]:
print(f" {r['name']}: " + ", ".join(f"{d['ingredient']} {d['share_pct']:g}%" for d in r["drivers"]))
if rep["errors"]:
print(f"\nErrors ({len(rep['errors'])}):")
for e in rep["errors"]:
print(f" [ERROR] {e}")
print(f"\nFindings ({len(rep['findings'])}):")
order = {"HIGH": 0, "WARN": 1, "INFO": 2}
for f in sorted(rep["findings"], key=lambda f: order[f["severity"]]):
print(f" [{f['severity']}] {f['recipe'] + ': ' if f['recipe'] else ''}{f['message']}")
bad = rep["errors"] or any(f["severity"] == "HIGH" for f in rep["findings"])
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))【中文说明】用于审查项目依赖许可证，结合软件交付方式识别缺失信息、许可证组合及潜在兼容风险。提供依赖清单、许可证资料和 SaaS、分发或内部使用场景，可获取整改优先级与声明清单；重要结论需由专业人士复核。 Audits the licenses of your project's dependencies for how you actually ship (SaaS, distributed app or library, or internal tool): normalizes license names to SPDX, resolves OR and AND expressions, flags missing, AGPL, GPL, source-available, and incompatible licenses, and writes a ranked fix plan and notices list. Includes a tested stdlib Python auditor.
---
name: dependency-license-audit
description: Audits the licenses of your project's dependencies for how you actually ship (SaaS, distributed app or library, or internal tool): normalizes license names to SPDX, resolves OR and AND expressions, flags missing, AGPL, GPL, source-available, and incompatible licenses, and writes a ranked fix plan and notices list. Includes a tested stdlib Python auditor.
---
# Dependency License Audit
You help engineering teams find license problems in their dependencies before a release, an acquisition review, or a customer security questionnaire finds them. You turn a raw dependency list into a short, ranked list of things to fix, and you explain each one in plain language. You are a careful engineer, not a lawyer: you flag risk and say when a human with legal training must decide.
## Files in this skill
- `scripts/audit_licenses.py` - normalizes and classifies every dependency license and rates it for the chosen distribution model (Python 3 standard library only)
- `references/license-categories.md` - what permissive, weak copyleft, strong copyleft, network copyleft, and source-available mean in practice, with common SPDX ids
- `references/remediation-playbook.md` - how to fix each kind of finding, from replacing a package to adding notices
- `templates/license-audit-report.md` - the report layout
- `examples/example-saas-audit.md` - a worked audit of a Node.js SaaS backend
## Workflow
### 1. Understand how the project is shipped
Ask, or infer and state as an assumption:
- Project license: proprietary (closed source) or an open source license (SPDX id).
- Distribution: `saas` (users only reach it over a network), `distributed` (mobile or desktop apps, on-prem installs, devices, SDKs, published libraries, container images given to customers), or `internal` (only employees use it).
- Any company policy: always-allowed and always-blocked licenses.
The same dependency can be fine for one model and a blocker for another, so never skip this step.
### 2. Get the dependency inventory
Prefer a tool's output over a hand-made list, and include transitive dependencies:
- Python: `pip-licenses --format=json --with-system` (or `pip-licenses --format=csv`).
- Node.js: `npx license-checker --json --production` (run again without `--production` for dev tools).
- Anything else: a CSV with `name,version,license,scope` exported from an SBOM or the package manager.
Mark build, test, and lint tools as `scope=dev`; they are not shipped, so their findings drop to INFO.
### 3. Run the audit
```bash
python3 scripts/audit_licenses.py deps.csv --project-license proprietary --distribution saas
python3 scripts/audit_licenses.py licenses.json --distribution distributed --deny AGPL-3.0-only
python3 scripts/audit_licenses.py deps.csv --json
```
The script prints severity counts, a table of every package with its normalized license and category, findings with reasons, version-to-version license changes, and the licenses that need text in your third-party notices. Exit code 1 means at least one HIGH finding.
If you cannot run the script, classify each license by hand with `references/license-categories.md` and say so in the report.
### 4. Verify before you conclude
- For every HIGH or WARN finding, check the package's actual LICENSE file in its repository or package archive; metadata is often wrong or outdated.
- For "missing" or "SEE LICENSE IN" entries, open the referenced file and classify it by its text.
- For dual-licensed packages (OR), record which option you choose.
- For packages whose license changed between versions, confirm which version is actually installed (lock file).
### 5. Recommend and report
Use `references/remediation-playbook.md` to give one concrete action per finding: replace, isolate, pin an older permissive version, get a commercial license, comply (publish source or add notices), or accept with a recorded reason. Fill in `templates/license-audit-report.md`, as in `examples/example-saas-audit.md`.
## Rules
- Never say a license "is fine" or "is illegal" as a legal conclusion. Say what the license usually requires and when to ask legal counsel.
- Do not guess a license from a package name or popularity. Unknown stays unknown until someone reads the license text.
- Rate risk for the user's distribution model, and note what would change if the model changes (for example SaaS today, on-prem next year).
- Keep the report short: blockers first, then obligations (notices, source offers), then the clean list as a count.
- Treat the audit as a snapshot; recommend running it in CI on every dependency change.
FILE:references/license-categories.md
# License categories in practice
Severity in the script depends on the category and on how the project is shipped. This page explains the categories in plain language. It is an engineering summary, not legal advice.
## Public domain and near public domain
SPDX: `0BSD`, `Unlicense`, `CC0-1.0`, `MIT-0`, `WTFPL`
- No meaningful obligations. Keeping a notice is still good practice.
## Permissive
SPDX: `MIT`, `ISC`, `BSD-2-Clause`, `BSD-3-Clause`, `Apache-2.0`, `Zlib`, `BSL-1.0` (Boost), `PSF-2.0`, `Python-2.0`, `CC-BY-4.0` (data such as caniuse-lite)
- Use, modify, and ship in closed source products.
- Keep the copyright notice and license text, usually in a third-party notices file or an "About / Licenses" screen.
- Apache-2.0 also requires keeping any NOTICE file, stating significant changes to its files, and includes a patent grant that ends if you sue over patents in that code.
- BSD-3-Clause forbids using the authors' names to promote your product.
## Weak copyleft (file or library level)
SPDX: `LGPL-2.1-*`, `LGPL-3.0-*`, `MPL-2.0`, `EPL-1.0`, `EPL-2.0`, `CDDL-1.0`, `CDDL-1.1`, GPL with a linking exception (for example `GPL-2.0-only WITH Classpath-exception-2.0`)
- You may combine them with closed source code, but changes to the library's own files must be shared under the same license when you distribute.
- LGPL: users must be able to replace the library with their own version. Dynamic linking (shared library, separate jar, npm package) normally makes this easy; static linking or bundling into one binary needs care.
- MPL-2.0: copyleft applies per file. Keep MPL files separate and publish changes to those files.
## Strong copyleft
SPDX: `GPL-2.0-only`, `GPL-2.0-or-later`, `GPL-3.0-only`, `GPL-3.0-or-later`
- If you distribute software that includes or links GPL code, the combined work must be offered under the GPL with its source code.
- Running GPL software on your own servers and letting users reach it over a network is not distribution, so pure SaaS use is usually allowed. Shipping an app, an on-prem installer, a device firmware, or a container image to customers is distribution.
- Version matters: `GPL-2.0-only` code cannot be combined with GPL-3.0 or Apache-2.0 code in one work. `-or-later` gives more room.
## Network copyleft
SPDX: `AGPL-3.0-only`, `AGPL-3.0-or-later`, `SSPL-1.0` (not OSI approved)
- Like the GPL, plus: if users interact with a modified version over a network, you must offer them its source code.
- For a closed source SaaS product this is the most common blocker. Even unmodified use triggers internal reviews at most companies.
- SSPL goes further: offering the software as a service can require releasing the source of your whole service stack.
## Source-available and non-commercial
SPDX or names: `BUSL-1.1` (Business Source License), `Elastic-2.0`, Commons Clause add-ons, `CC-BY-NC-*`, `PolyForm-Noncommercial-1.0.0`
- Source is visible, but use is restricted: often no competing hosted service, no production use above a limit, or no commercial use at all, sometimes until a change date.
- Always needs a review of the exact terms. Many of these switch to an open license after a few years, so an older or newer version may differ.
## Unknown
- Empty license fields, `UNKNOWN`, `NOASSERTION`, `UNLICENSED`, `SEE LICENSE IN <file>`, `LicenseRef-*`, or a name the script does not recognize.
- "No license" means all rights reserved: you have no permission to use it. Find the real terms or replace the package.
## Expressions
- `MIT OR Apache-2.0`: you may choose either; pick the one that suits you and record the choice.
- `MIT AND BSD-3-Clause`: you must follow both; the strictest one sets the category.
- `GPL-2.0-only WITH <exception>`: the exception narrows the copyleft; read it.
FILE:references/remediation-playbook.md
# Remediation playbook
Pick one action per finding and write it down with an owner. Prefer the cheapest action that removes the risk.
## Order of work
1. Unknown and missing licenses (you may have no right to use the code at all).
2. Network copyleft and source-available packages in shipped or hosted code.
3. Strong copyleft in distributed products, and version incompatibilities in open source projects.
4. Weak copyleft obligations (linking, publishing changes).
5. Notices for everything else.
## Actions
### Find the real license
- Open the package's repository and archive, read LICENSE, COPYING, or the header of the main source file.
- Check whether the metadata is just stale (a common case for old packages and forks).
- If there is truly no license, ask the maintainer to add one (link to choosealicense.com in the request) or replace the package.
### Replace the package
- Search for a maintained alternative with a permissive license and similar API.
- Estimate effort in hours and add it to the backlog with the release it blocks.
- For transitive dependencies, look for the direct dependency that pulls it in; a different version or a configuration flag often drops it.
### Pin a version under the old license
- When a project switched from a permissive license to source-available (or the reverse), the old versions keep their old license.
- Pinning is a short-term fix: no security updates. Schedule a proper replacement.
### Isolate it
- Run the copyleft tool as a separate process or service that talks to your code over a command line, HTTP, or a queue, instead of linking it into your code.
- Keep it unmodified if possible. Document the boundary.
- This is a common way to use GPL command line tools, but have legal counsel confirm it for AGPL and SSPL.
### Buy a commercial license
- Many dual-licensed projects (open source plus commercial) sell a license that removes copyleft obligations. Note the price and renewal terms.
### Comply
- Weak copyleft: keep the library as a separate, replaceable file or package; publish any changes to its own files; include its license.
- GPL in a distributed product: only if the whole product can be open source under compatible terms. This is a business decision, not an engineering one.
- AGPL in a hosted service: offer the corresponding source to users of the service, including your changes.
### Accept with a recorded reason
- Dev-only tools (linters, test runners, build tools) that never ship are usually fine. Record "dev only, not distributed".
- Internal-only use of GPL or AGPL tools is usually fine. Record who uses it and that it is not exposed to outside users.
## Third-party notices
- Generate a notices file from the inventory: package, version, license, copyright line, full license text (one copy per distinct license text).
- Include Apache NOTICE files as they are.
- Ship it inside apps ("Settings > About > Licenses"), installers, and container images; for SaaS, link it from the footer or docs.
## Keep it clean
- Run the audit in CI on every change to a lock file and fail the build on HIGH.
- Keep an allow list (for example MIT, BSD, ISC, Apache-2.0) and a deny list (for example AGPL-3.0, SSPL-1.0, BUSL-1.1) agreed with legal counsel, and pass them with `--allow` and `--deny`.
FILE:templates/license-audit-report.md
# Dependency License Audit: <project name>
Date: <YYYY-MM-DD>
Project license: <proprietary | SPDX id>
Distribution: <saas | distributed | internal> (<one line: how users get the software>)
Inventory source: <tool and command, lock file commit>
Packages audited: <n> (<runtime n>, <dev n>)
## Summary
- Blockers (HIGH): <n> - <one line naming them>
- Obligations (WARN): <n> - <one line>
- Notes (INFO): <n>
- Clean (OK): <n>
- Verdict: <ready to ship | ship after fixes | blocked> because <reason>.
## Blockers
| Package | Version | License (as found) | Category | Why it matters for us | Action | Owner | Due |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## Obligations
| Package | License | What we must do | Done? |
|---|---|---|---|
| | | | |
## Verified by hand
- <package>: metadata said <x>, LICENSE file says <y>.
## Decisions and assumptions
- Dual-licensed packages and the option chosen: <package: option>
- Accepted risks with reason: <package: reason, approved by>
- Assumptions: <for example "the admin dashboard is internal only">
## What changes if we change how we ship
- <for example: "If we ship an on-prem version, readline-sync (GPL-3.0) becomes a blocker.">
## Third-party notices
Licenses that need full text in the notices file: <list>
Notices file location: <path or URL>
## Next audit
Run in CI on every lock file change; full review before <next release or date>.
_This report is an engineering triage, not legal advice. Blockers must be confirmed by legal counsel before relying on any exception._
FILE:examples/example-saas-audit.md
# Example: license audit of a Node.js SaaS backend
## The request
"We're a closed source B2B SaaS. Enterprise customers keep sending security questionnaires asking about open source licenses, and next year we may sell an on-prem version. Here is our dependency export. Anything we should worry about?"
Inventory (`deps.csv`, exported from the lock file, 18 rows):
```
name,version,license,scope
express,4.19.2,MIT,runtime
lodash,4.17.21,MIT,runtime
pg,8.12.0,MIT,runtime
sharp,0.33.4,Apache-2.0,runtime
dompurify,3.1.6,(MPL-2.0 OR Apache-2.0),runtime
ghostscript-wrapper,2.1.0,AGPL-3.0-only,runtime
chart-kit-pro,1.4.0,SEE LICENSE IN LICENSE.md,runtime
readline-sync,1.4.10,GPL-3.0,runtime
mariadb-connector,3.3.1,LGPL-2.1-or-later,runtime
fast-csv-lite,0.9.0,,runtime
redis-om,0.4.3,Business Source License 1.1,runtime
jest,29.7.0,MIT,dev
eslint-plugin-gpl-thing,1.0.2,GPL-2.0-only,dev
argparse,2.0.1,Python-2.0,runtime
tslib,2.6.3,0BSD,runtime
lodash,3.10.1,MIT,runtime
search-client,7.10.2,Apache-2.0,runtime
search-client,8.1.0,Elastic License 2.0,runtime
```
## Script run
```
$ python3 scripts/audit_licenses.py deps.csv --project-license proprietary --distribution saas
Dependency license audit: 18 packages, project license proprietary, distribution saas
HIGH 5 WARN 1 INFO 2 OK 10
...
HIGH chart-kit-pro@1.4.0 SEE LICENSE IN LICENSE.md unknown runtime
HIGH fast-csv-lite@0.9.0 (missing) unknown runtime
HIGH redis-om@0.4.3 BUSL-1.1 source-available runtime
HIGH search-client@8.1.0 Elastic-2.0 source-available runtime
HIGH ghostscript-wrapper@2.1.0 AGPL-3.0-only network-copyleft runtime
WARN readline-sync@1.4.10 GPL-3.0-only strong-copyleft runtime
INFO eslint-plugin-gpl-thing@1.0.2 GPL-2.0-only strong-copyleft dev
INFO mariadb-connector@3.3.1 LGPL-2.1-or-later weak-copyleft runtime
...
Notes:
- search-client: different licenses across versions (Apache-2.0, Elastic-2.0); check which version you ship.
```
Exit code 1 (HIGH findings).
## The report (filled template, shortened)
# Dependency License Audit: acme-api
Project license: proprietary. Distribution: saas (customers use the web app and REST API; nothing is installed on their machines today).
Packages audited: 18 (16 runtime, 2 dev).
### Summary
- Blockers (HIGH): 5 - two packages with no usable license, two source-available packages, one AGPL package.
- Obligations (WARN): 1 - a GPL-3.0 package that is fine for SaaS but blocks the planned on-prem version.
- Verdict: ship after fixes. Nothing here is unusual, and each item has a cheap fix.
### Blockers
| Package | License (as found) | Why it matters for us | Action |
|---|---|---|---|
| chart-kit-pro 1.4.0 | SEE LICENSE IN LICENSE.md | The LICENSE.md is a commercial license that allows use only with a paid key. We have no key on record. | Confirm with the team that bought it, or replace with a permissive charting library. |
| fast-csv-lite 0.9.0 | (missing) | No license means no permission to use. | Ask the maintainer to add a license; meanwhile switch to a maintained CSV parser with MIT terms. |
| redis-om 0.4.3 | BUSL-1.1 | Business Source License limits production use until its change date; terms vary per project. | Read the exact "Additional Use Grant". If it forbids our use, replace or buy a license. |
| search-client 8.1.0 | Elastic-2.0 | Elastic License 2.0 forbids offering the software as a managed service. Version 7.10.2 in the same tree is Apache-2.0. | Check the lock file: which modules import 8.1.0? Pin both to the Apache-2.0 line or move to an Apache-licensed fork. |
| ghostscript-wrapper 2.1.0 | AGPL-3.0-only | We render customer PDFs with it inside the API; AGPL network use requires offering source to users. | Move PDF rendering to a separate, unmodified worker service and have legal counsel confirm, or buy the commercial license. |
### Obligations
| Package | License | What we must do |
|---|---|---|
| readline-sync 1.4.10 | GPL-3.0-only | Fine for SaaS. It becomes a blocker if the on-prem version ships with it. It is only used by an admin CLI script, so remove it from the server image. |
| mariadb-connector 3.3.1 | LGPL-2.1-or-later | Keep it as a normal npm dependency (replaceable); publish changes if we ever patch it. |
### Verified by hand
- dompurify: dual licensed `MPL-2.0 OR Apache-2.0`; we choose Apache-2.0.
- eslint-plugin-gpl-thing: dev only, never shipped. Accepted.
### What changes if we ship on-prem
- readline-sync (GPL-3.0) becomes HIGH, mariadb-connector (LGPL) needs the replaceability check, and Apache-2.0 packages need their NOTICE files in the installer. Rerun with `--distribution distributed` before that release.
### Third-party notices
Licenses that need full text: Apache-2.0, LGPL-2.1-or-later, MIT, Python-2.0. Link the notices page from the app footer.
_Engineering triage, not legal advice._
## Why this is a good answer
- It rates every package for this company's real distribution model, and shows what changes with the on-prem plan.
- It checks the facts the metadata cannot show (the commercial LICENSE.md, the duplicate search-client versions).
- Every blocker has one concrete, cheap action.
FILE:scripts/audit_licenses.py
#!/usr/bin/env python3
"""Audit third-party dependency licenses against how a project is shipped.
Usage:
python3 audit_licenses.py deps.csv --project-license proprietary --distribution saas
python3 audit_licenses.py licenses.json --distribution distributed [--json]
cat deps.csv | python3 audit_licenses.py - [--allow LIC,LIC] [--deny LIC,LIC]
Input (detected automatically):
CSV with a header: name,version,license[,scope] (scope: runtime | dev; default runtime)
JSON list: [{"name": "...", "version": "...", "license": "...", "scope": "dev"}]
pip-licenses --format=json: [{"Name": "...", "Version": "...", "License": "..."}]
license-checker --json (npm): {"pkg@1.2.3": {"licenses": "MIT"}, ...}
Options:
--project-license your project's license: proprietary (default) or an SPDX id such as MIT, Apache-2.0, GPL-3.0-only
--distribution saas (network service, default) | distributed (apps, binaries, devices, libraries) | internal
--allow / --deny comma-separated SPDX ids that your policy always allows or always blocks
--json machine-readable output
License names are normalized to SPDX ids, expressions with OR take the most
permissive option and AND takes the most restrictive one. Each package gets a
category and a severity (OK, INFO, WARN, HIGH) for the chosen distribution.
Dev-only packages are not shipped, so their findings drop to INFO.
Exit code: 0 no HIGH findings, 1 at least one HIGH, 2 usage or input error.
This is an engineering triage aid, not legal advice. Standard library only.
"""
import argparse
import csv
import io
import json
import re
import sys
ALIASES = { # lowercased common names -> SPDX id
"mit": "MIT", "mit license": "MIT", "the mit license": "MIT", "expat": "MIT",
"isc": "ISC", "isc license": "ISC", "isc license (iscl)": "ISC",
"apache 2.0": "Apache-2.0", "apache-2": "Apache-2.0", "apache 2": "Apache-2.0", "apache2": "Apache-2.0",
"apache license 2.0": "Apache-2.0", "apache license, version 2.0": "Apache-2.0",
"apache software license": "Apache-2.0", "apache license version 2.0": "Apache-2.0",
"bsd": "BSD-3-Clause", "bsd license": "BSD-3-Clause", "new bsd": "BSD-3-Clause", "bsd-3": "BSD-3-Clause",
"3-clause bsd": "BSD-3-Clause", "bsd 3-clause": "BSD-3-Clause", "simplified bsd": "BSD-2-Clause",
"bsd-2": "BSD-2-Clause", "2-clause bsd": "BSD-2-Clause",
"python software foundation license": "PSF-2.0", "psf": "PSF-2.0", "psfl": "PSF-2.0",
"mozilla public license 2.0 (mpl 2.0)": "MPL-2.0", "mpl 2.0": "MPL-2.0", "mpl-2": "MPL-2.0", "mpl2": "MPL-2.0",
"gplv2": "GPL-2.0-only", "gpl-2.0": "GPL-2.0-only", "gpl v2": "GPL-2.0-only", "gpl2": "GPL-2.0-only",
"gplv2+": "GPL-2.0-or-later", "gpl-2.0+": "GPL-2.0-or-later",
"gplv3": "GPL-3.0-only", "gpl-3.0": "GPL-3.0-only", "gpl v3": "GPL-3.0-only", "gpl3": "GPL-3.0-only",
"gplv3+": "GPL-3.0-or-later", "gpl-3.0+": "GPL-3.0-or-later", "gpl": "GPL-3.0-or-later",
"gnu general public license v2 (gplv2)": "GPL-2.0-only", "gnu general public license v3 (gplv3)": "GPL-3.0-only",
"gnu general public license v2 or later (gplv2+)": "GPL-2.0-or-later",
"gnu general public license v3 or later (gplv3+)": "GPL-3.0-or-later",
"lgpl": "LGPL-2.1-or-later", "lgplv2": "LGPL-2.1-only", "lgplv2+": "LGPL-2.1-or-later", "lgpl-2.1": "LGPL-2.1-only",
"lgplv3": "LGPL-3.0-only", "lgplv3+": "LGPL-3.0-or-later", "lgpl-3.0": "LGPL-3.0-only",
"gnu lesser general public license v2 (lgplv2)": "LGPL-2.1-only",
"gnu lesser general public license v3 (lgplv3)": "LGPL-3.0-only",
"gnu lesser general public license v2 or later (lgplv2+)": "LGPL-2.1-or-later",
"agpl": "AGPL-3.0-only", "agplv3": "AGPL-3.0-only", "agpl-3.0": "AGPL-3.0-only",
"gnu affero general public license v3": "AGPL-3.0-only", "gnu affero general public license v3 (agplv3)": "AGPL-3.0-only",
"eclipse public license 2.0": "EPL-2.0", "epl 2.0": "EPL-2.0", "epl-2": "EPL-2.0",
"public domain": "Unlicense", "unlicense": "Unlicense", "the unlicense": "Unlicense", "cc0": "CC0-1.0",
"boost software license 1.0": "BSL-1.0", "zlib license": "Zlib", "zlib": "Zlib",
"business source license 1.1": "BUSL-1.1", "elastic license 2.0": "Elastic-2.0",
"server side public license": "SSPL-1.0", "sspl": "SSPL-1.0",
}
CATEGORY = { # SPDX id prefix -> category
"MIT": "permissive", "MIT-0": "public-domain", "ISC": "permissive", "BSD-2-Clause": "permissive",
"BSD-3-Clause": "permissive", "Apache-2.0": "permissive", "PSF-2.0": "permissive", "Python-2.0": "permissive",
"Zlib": "permissive", "BSL-1.0": "permissive", "X11": "permissive", "BlueOak-1.0.0": "permissive",
"CC-BY-4.0": "permissive", "CC-BY-3.0": "permissive", "Unicode-DFS-2016": "permissive", "Unicode-3.0": "permissive",
"0BSD": "public-domain", "Unlicense": "public-domain", "CC0-1.0": "public-domain", "WTFPL": "public-domain",
"MPL-2.0": "weak-copyleft", "EPL-1.0": "weak-copyleft", "EPL-2.0": "weak-copyleft", "CDDL-1.0": "weak-copyleft",
"CDDL-1.1": "weak-copyleft", "LGPL-2.1": "weak-copyleft", "LGPL-3.0": "weak-copyleft", "CC-BY-SA-4.0": "weak-copyleft",
"GPL-2.0": "strong-copyleft", "GPL-3.0": "strong-copyleft",
"AGPL-3.0": "network-copyleft", "SSPL-1.0": "network-copyleft",
"BUSL-1.1": "source-available", "Elastic-2.0": "source-available", "Commons-Clause": "source-available",
"CC-BY-NC-4.0": "source-available", "CC-BY-NC-SA-4.0": "source-available", "PolyForm-Noncommercial-1.0.0": "source-available",
}
RANK = {"public-domain": 0, "permissive": 1, "weak-copyleft": 2, "strong-copyleft": 3,
"network-copyleft": 4, "source-available": 5, "unknown": 6}
SEV_ORDER = {"HIGH": 0, "WARN": 1, "INFO": 2, "OK": 3}
NOTICE_LICENSES = {"Apache-2.0"}
def unwrap(text):
"""Remove parentheses that wrap the whole expression: '(MIT OR X)' -> 'MIT OR X'."""
t = text.strip()
while t.startswith("(") and t.endswith(")"):
depth = 0
for i, ch in enumerate(t):
depth += ch == "("
depth -= ch == ")"
if depth == 0 and i < len(t) - 1:
return t
t = t[1:-1].strip()
return t
def normalize(token):
t = unwrap(token)
if not t:
return ""
key = t.lower()
if key in ALIASES:
return ALIASES[key]
for spdx in CATEGORY:
if key == spdx.lower() or key.startswith(spdx.lower() + "-") or key == spdx.lower() + "+":
return t if key.startswith(spdx.lower() + "-") else spdx
return t
def category_of(spdx):
if spdx.upper().startswith("LICENSEREF") or spdx.upper().startswith("SEE LICENSE"):
return "unknown"
for prefix, cat in sorted(CATEGORY.items(), key=lambda kv: -len(kv[0])):
if spdx == prefix or spdx.startswith(prefix + "-") or spdx.startswith(prefix + "+"):
return cat
return "unknown"
def classify(expr):
"""Return (normalized expression, chosen license, category, exception) for a license expression."""
raw = (expr or "").strip()
if not raw or raw.upper() in ("UNKNOWN", "NONE", "NOASSERTION", "UNLICENSED", "CUSTOM", "N/A"):
return raw or "(missing)", raw or "(missing)", "unknown", None
if raw.upper().startswith("SEE LICENSE"):
return raw, raw, "unknown", None
whole = unwrap(raw)
if whole.lower() in ALIASES: # long names such as "... v2 or later (LGPLv2+)" are one license
spdx = ALIASES[whole.lower()]
return spdx, spdx, category_of(spdx), None
text = re.sub(r"\s+", " ", whole.replace("/", " OR ").replace(";", " OR "))
exception = None
m = re.search(r"\bWITH\s+([\w.+-]+)", text, re.I)
if m:
exception = m.group(1)
text = text[:m.start()] + text[m.end():]
if re.search(r"\sor\s", text, re.I) and not re.search(r"\sand\s", text, re.I):
parts, mode = re.split(r"\s+or\s+", text, flags=re.I), "or"
elif re.search(r"\sand\s", text, re.I) and not re.search(r"\sor\s", text, re.I):
parts, mode = re.split(r"\s+and\s+", text, flags=re.I), "and"
elif re.search(r"\s(or|and)\s", text, re.I):
parts, mode = re.split(r"\s+(?:or|and)\s+", text, flags=re.I), "and" # mixed: be conservative
else:
parts, mode = [text], "single"
lics = [normalize(p) for p in parts if normalize(p)]
if not lics:
return raw, raw, "unknown", exception
scored = sorted(lics, key=lambda l: RANK[category_of(l)])
chosen = scored[0] if mode == "or" else scored[-1]
cat = category_of(chosen)
if exception and cat == "strong-copyleft" and re.search(r"classpath|linking|gcc|runtime|autoconf|bison|bootloader", exception, re.I):
cat = "weak-copyleft"
joiner = " OR " if mode == "or" else " AND "
return joiner.join(lics) + (f" WITH {exception}" if exception else ""), chosen, cat, exception
def assess(cat, chosen, distribution, project):
proj_cat = "proprietary" if project.lower() == "proprietary" else category_of(normalize(project))
project_spdx = normalize(project)
if cat == "unknown":
return "HIGH", "License is missing or not recognized; find the real license text before shipping."
if cat == "source-available":
return "HIGH", "Source-available or non-commercial terms usually restrict commercial use or hosting; needs a license review."
if project_spdx.startswith("GPL-2.0-only") and chosen.startswith("Apache-2.0"):
return "HIGH", "Apache-2.0 is generally considered incompatible with GPL-2.0-only projects."
gpl3_family = re.match(r"(A|L)?GPL-3\.0", chosen)
if project_spdx.startswith("GPL-2.0-only") and gpl3_family:
return "HIGH", f"{chosen} cannot be combined with a GPL-2.0-only project; the project would need GPL-3.0 terms."
if re.match(r"GPL-3\.0|AGPL-3\.0", project_spdx) and chosen.startswith("GPL-2.0-only"):
return "HIGH", "GPL-2.0-only code cannot be relicensed under GPL-3.0; ask upstream for 'or later' terms or replace it."
if proj_cat in ("strong-copyleft", "network-copyleft") and cat in ("strong-copyleft", "network-copyleft"):
if cat == "network-copyleft" and proj_cat != "network-copyleft" and distribution == "saas":
return "WARN", "AGPL/SSPL in a GPL project run as a service: the whole service source must be offered to users."
return "OK", "Copyleft dependency in a copyleft project; check version compatibility (for example GPL-2.0-only vs GPL-3.0)."
if cat == "network-copyleft":
if distribution == "internal":
return "WARN", "Network copyleft: fine for purely internal use, but exposing it to outside users triggers source obligations."
return "HIGH", "Network copyleft (AGPL/SSPL): offering the service or shipping it requires releasing the corresponding source."
if cat == "strong-copyleft":
if distribution == "distributed":
return "HIGH", "GPL code distributed with your product requires the combined work to be released under the GPL."
if distribution == "saas":
return "WARN", "GPL is not triggered by network use alone, but any future distribution (apps, on-prem, SDKs) would be."
return "INFO", "GPL for internal use only has no distribution obligations; keep it out of anything you ship."
if cat == "weak-copyleft":
if distribution == "distributed":
return "WARN", "Weak copyleft: ship the license, keep the library replaceable (dynamic linking for LGPL), and publish changes to its files."
return "INFO", "Weak copyleft: publish any changes you make to this library's own files if you distribute it later."
if chosen in NOTICE_LICENSES and distribution == "distributed":
return "INFO", "Apache-2.0: include the license and any NOTICE file in your distribution."
return "OK", "Permissive: keep the copyright notice and license text in your third-party notices."
def load(text):
stripped = text.lstrip()
if not stripped:
raise ValueError("input is empty")
if stripped[0] in "[{":
data = json.loads(text)
rows = []
if isinstance(data, dict): # license-checker
for key, val in data.items():
name, _, ver = key.rpartition("@")
if not name:
name, ver = key, ""
lic = val.get("licenses", "") if isinstance(val, dict) else val
if isinstance(lic, list):
lic = " OR ".join(lic)
rows.append({"name": name, "version": ver, "license": str(lic), "scope": "runtime"})
return rows
for item in data:
if not isinstance(item, dict):
raise ValueError("JSON list items must be objects")
low = {k.lower(): v for k, v in item.items()}
lic = low.get("license", low.get("licenses", ""))
if isinstance(lic, list):
lic = " OR ".join(lic)
rows.append({"name": str(low.get("name", "")), "version": str(low.get("version", "")),
"license": str(lic or ""), "scope": str(low.get("scope", "runtime") or "runtime").lower()})
return rows
reader = csv.DictReader(io.StringIO(text))
fields = [f.strip().lower() for f in (reader.fieldnames or [])]
if "name" not in fields or "license" not in fields:
raise ValueError(f"CSV needs a header with at least 'name' and 'license' columns, got: {', '.join(fields) or '(none)'}")
rows = []
for r in reader:
r = {(k or "").strip().lower(): (v or "").strip() for k, v in r.items()}
if not r.get("name"):
continue
rows.append({"name": r["name"], "version": r.get("version", ""), "license": r.get("license", ""),
"scope": (r.get("scope") or "runtime").lower()})
return rows
def audit(rows, args):
allow = {normalize(x) for x in args.allow.split(",") if x.strip()} if args.allow else set()
deny = {normalize(x) for x in args.deny.split(",") if x.strip()} if args.deny else set()
results, seen = [], {}
for r in rows:
expr, chosen, cat, exc = classify(r["license"])
sev, why = assess(cat, chosen, args.distribution, args.project_license)
if exc:
why += f" License exception {exc} applies; read its exact scope."
if chosen in deny:
sev, why = "HIGH", f"{chosen} is on your deny list."
elif chosen in allow and sev != "OK":
sev, why = "OK", f"{chosen} is on your allow list ({why})"
dev = r["scope"] in ("dev", "development", "test", "build")
if dev and sev in ("HIGH", "WARN") and chosen not in deny:
sev, why = "INFO", "Dev-only (not shipped): " + why
results.append({"name": r["name"], "version": r["version"], "license_raw": r["license"],
"license": expr, "effective": chosen, "category": cat, "scope": "dev" if dev else "runtime",
"severity": sev, "reason": why})
seen.setdefault(r["name"].lower(), set()).add(expr)
notes = []
for name, lics in seen.items():
if len(lics) > 1:
notes.append(f"{name}: different licenses across versions ({', '.join(sorted(lics))}); check which version you ship.")
results.sort(key=lambda x: (SEV_ORDER[x["severity"]], -RANK[x["category"]], x["name"].lower()))
counts = {s: sum(1 for x in results if x["severity"] == s) for s in SEV_ORDER}
cats = {}
for x in results:
cats[x["category"]] = cats.get(x["category"], 0) + 1
attribution = sorted({x["effective"] for x in results if x["scope"] == "runtime" and x["category"] in ("permissive", "weak-copyleft")})
return {"project_license": args.project_license, "distribution": args.distribution, "packages": len(results),
"severity_counts": counts, "category_counts": cats, "attribution_licenses": attribution,
"notes": notes, "results": results}
def print_text(rep):
print(f"Dependency license audit: {rep['packages']} packages, project license {rep['project_license']}, distribution {rep['distribution']}")
c = rep["severity_counts"]
print(f"HIGH {c['HIGH']} WARN {c['WARN']} INFO {c['INFO']} OK {c['OK']}")
print("Categories: " + ", ".join(f"{k} {v}" for k, v in sorted(rep["category_counts"].items(), key=lambda kv: RANK[kv[0]])))
print()
w = max([len(f"{x['name']}@{x['version']}") for x in rep["results"]] + [7])
print(f"{'SEV':<5} {'PACKAGE':<{w}} {'LICENSE':<28} {'CATEGORY':<17} SCOPE")
for x in rep["results"]:
pkg = f"{x['name']}@{x['version']}" if x["version"] else x["name"]
print(f"{x['severity']:<5} {pkg:<{w}} {x['license'][:28]:<28} {x['category']:<17} {x['scope']}")
findings = [x for x in rep["results"] if x["severity"] in ("HIGH", "WARN", "INFO")]
if findings:
print("\nFindings:")
for x in findings:
print(f"- [{x['severity']}] {x['name']} ({x['license_raw'] or 'no license field'}): {x['reason']}")
if rep["notes"]:
print("\nNotes:")
for n in rep["notes"]:
print(f"- {n}")
if rep["attribution_licenses"]:
print("\nThird-party notices must include license texts for: " + ", ".join(rep["attribution_licenses"]))
print("\nTriage aid only, not legal advice.")
def main():
p = argparse.ArgumentParser(description="Audit dependency licenses.", add_help=True)
p.add_argument("path", help="CSV or JSON file, or - for stdin")
p.add_argument("--project-license", default="proprietary")
p.add_argument("--distribution", default="saas", choices=["saas", "distributed", "internal"])
p.add_argument("--allow", default="")
p.add_argument("--deny", default="")
p.add_argument("--json", action="store_true")
args = p.parse_args()
try:
text = sys.stdin.read() if args.path == "-" else open(args.path, encoding="utf-8").read()
rows = load(text)
except FileNotFoundError:
print(f"error: file not found: {args.path}", file=sys.stderr)
return 2
except (ValueError, json.JSONDecodeError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
if not rows:
print("error: no dependencies found in the input", file=sys.stderr)
return 2
rep = audit(rows, args)
if args.json:
print(json.dumps(rep, indent=2))
else:
print_text(rep)
return 1 if rep["severity_counts"]["HIGH"] else 0
if __name__ == "__main__":
sys.exit(main())【中文说明】用于检查 Markdown 文档的无障碍问题，包括图片替代文本、标题层级、链接措辞和表格表头等。提供 README 或其他文档内容，并附图片背景说明，要求 AI 返回问题清单、合适的替代文本或修正后的文件。 Checks READMEs and Markdown docs for accessibility: missing or file-name alt text, heading levels, vague or duplicate link text, bare URLs, empty table headers, color-only instructions, ALL CAPS, and emoji bullets, then writes purposeful alt text and a corrected file. Includes a tested stdlib Python checker for CI.
---
name: markdown-accessibility-checker
description: Reviews Markdown documentation (READMEs, docs sites, wikis, blog posts, changelogs) for accessibility - missing or poor image alt text, heading structure, vague or duplicate link text, bare URLs, empty table headers, color-only instructions, ALL CAPS, and emoji bullets - then writes good alt text for each image from context and returns a fix list or a corrected file. Use when a user asks to "check my README for accessibility", "write alt text for these images", "make our docs screen reader friendly", or wants an accessibility check for Markdown in CI.
---
# Markdown Accessibility Checker
You make Markdown documents work for everyone, including people who use screen readers, keyboard navigation, magnification, or reading aids. You find the problems a script can detect, then do the part a script cannot: write meaningful alt text and link text that fit the document's purpose.
## Files in this skill
- `scripts/check_md_a11y.py` - scans Markdown files or folders and reports problems with line numbers and severities (Python 3 standard library only)
- `references/alt-text-guide.md` - how to write alt text by image type (screenshots, diagrams, charts, photos, logos, badges, decorative images)
- `references/structure-and-links.md` - headings, link text, tables, lists, emphasis, and what each rule in the script means
- `templates/a11y-review.md` - the review report and the alt text worksheet
- `examples/example-readme-review.md` - a worked review of a project README
## Workflow
### 1. Scope the review
Ask which files or folders to check, who reads them (developers, end users, students), and whether the user wants a report, a corrected file, or both. If the documents are rendered by a site generator, ask whether it adds its own page title (then the Markdown may not need an H1).
### 2. Run the checker
```bash
python3 scripts/check_md_a11y.py README.md docs/
python3 scripts/check_md_a11y.py docs/ --json
python3 scripts/check_md_a11y.py docs/ --fail-on warn # for CI
```
It ignores fenced code blocks and inline code. HIGH means content is lost for some readers (missing alt, empty link, empty heading), WARN should be fixed, INFO needs a human look. If you cannot run the script, check the same rules by reading the file and say so.
### 3. Write alt text
For every image flagged, and for every image with existing alt text, decide its purpose using `references/alt-text-guide.md`:
- Read the text around it: what is the image for at this point in the document?
- If you can see the image, describe what matters for that purpose. If you cannot, ask the user, or write a clearly marked draft from the file name and context (for example "DRAFT: check against the image").
- Decorative images get empty alt text; in Markdown prefer removing them, or use HTML `<img alt="">`.
Fill in the alt text worksheet in `templates/a11y-review.md`.
### 4. Fix structure and links
Apply `references/structure-and-links.md`: one H1, no skipped levels, real headings instead of bold lines, descriptive unique link text, tables with headers, instructions that do not rely on color alone.
### 5. Deliver
- Report: fill in `templates/a11y-review.md`, as in `examples/example-readme-review.md`, with the most important fixes first.
- Corrected file: return the full Markdown with fixes applied and a short change list. Do not change meaning, code samples, or links' destinations.
- Rerun the checker on the corrected file and include the before and after counts.
## Rules
- Never invent what an image shows. If you have not seen it, mark the alt text as a draft.
- Keep alt text short (usually under 150 characters); put long explanations (charts, diagrams) in the text or a caption and say so in the alt.
- Do not start alt text with "image of" or "picture of".
- Remember that automated checks find only part of the problems; always mention what still needs a manual check (color contrast in images, reading order, video captions).
- Respect the author's voice; fix accessibility, not style.
FILE:references/alt-text-guide.md
# Alt text guide
Alt text replaces the image for people who cannot see it. Ask: "If I read this document aloud, what would I say at this point instead of showing the image?" The answer depends on why the image is there, not only on what is in it.
## Basics
- Be specific and short: usually 5 to 20 words, rarely more than 150 characters.
- Lead with the important part. Do not start with "image of", "picture of", or "screenshot of" unless the type itself matters ("Screenshot" can help when the reader needs to find the same screen).
- End with a period so screen readers pause.
- Do not repeat the caption or the sentence next to the image word for word.
- Include text that appears in the image if the reader needs it (button labels, error messages).
## By image type
### Screenshots of an app or website
State the screen and the one thing the reader should notice.
- Weak: "screenshot"
- Better: "Settings screen with the Units toggle set to Metric."
### Hero images and banners
If it only sets the mood, it is decorative. If it shows the product, describe it in one sentence.
- "Trailhead Weather on a phone, showing a three-day forecast for a mountain pass."
### Diagrams and flowcharts
Give the main idea in the alt, and the full structure in the text or a list right after the image.
- Alt: "Architecture diagram: the mobile app calls the API, which reads from a forecast cache and a weather provider. Details in the list below."
### Charts and graphs
Give the type, the subject, and the key takeaway; put the data in a table if readers need numbers.
- "Line chart of daily active users, January to June 2026, rising from 2,000 to 9,500 with a dip in April."
### Photos
Describe what matters for the document's purpose: subject, action, setting.
- "Snow-covered ridge trail with orange marker poles leading to a hut."
### Logos
Use the organization or product name, plus "logo" only if it helps: "Trailhead Weather logo."
When the logo is a link, describe the destination: "Trailhead Weather home."
### Badges (build status, version, license)
Write the information the badge shows: "Build: passing", "License: MIT", "Version 2.4.1".
### Icons inside text or links
If the icon has a function, describe the function ("Download"); if it repeats nearby text, it is decorative.
### Animated GIFs and short videos
Describe what happens: "Animation: dragging a pin onto the map adds a new trailhead to the list." Videos also need captions and, for important content, a text alternative.
### Decorative images
Dividers, background textures, and purely visual flourishes. In HTML use `alt=""` so screen readers skip them. In Markdown, `` is ambiguous (often a mistake), so prefer removing the image or using `<img src="divider.png" alt="">`.
## Common mistakes
- File names as alt text: "IMG_2041.jpg", "hero-final-v3".
- One word: "image", "logo", "chart".
- Keyword stuffing for search engines.
- Different alt text for the same image used in several places.
- Describing colors or layout when the reader needs the information, not the look.
FILE:references/structure-and-links.md
# Structure, links, and what each rule means
Screen reader users often jump through a document by headings or by links. Good structure is what makes that possible.
## Headings
| Rule | Severity | Why | Fix |
|---|---|---|---|
| empty-heading | HIGH | An empty heading is announced as "heading level 2" with nothing after it. | Add text or remove the #. |
| no-h1 | WARN | The page has no title in the heading outline. | Start with one `# Title`. Skip this if the site generator adds the H1 from front matter. |
| multiple-h1 | WARN | Several titles confuse the outline. | Keep one H1; use `##` for sections. |
| heading-skip | WARN | Jumping from H2 to H4 suggests a missing section. | Use the next level down (H3). Pick levels for structure, not for font size. |
| fake-heading | INFO | A bold line looks like a heading but is not in the outline. | Use a real heading if it starts a section; keep bold if it is just emphasis. |
| no-headings | INFO | Long text without headings is hard to scan. | Add section headings every few paragraphs. |
## Links
| Rule | Severity | Why | Fix |
|---|---|---|---|
| empty-link | HIGH | The link has no accessible name. | Add descriptive text. |
| vague-link | WARN | "Click here" or "read more" means nothing in a list of links. | Name the destination: "download the latest release", "setup guide". |
| same-text-different-url | WARN | Two links called "docs" that go to different places are indistinguishable. | Make each text unique: "API reference", "setup guide". |
| url-as-link-text | INFO | Long URLs are read character by character. | Use a short description; keep the URL only if readers must copy it. |
| bare-url | INFO | Same as above, and not always clickable. | Wrap in a descriptive link. |
Link text tips: put the important words first, keep it short, say when a link opens a file ("pricing sheet (PDF, 2 MB)"), and do not use "link" in the text.
## Images
| Rule | Severity | Why |
|---|---|---|
| img-missing-alt | HIGH | `` gives no alternative; some readers announce the file name. |
| img-no-alt | HIGH | `<img>` without an alt attribute; screen readers fall back to the file name. |
| img-generic-alt | WARN | "image", "screenshot", "logo" alone carry no information. |
| img-filename-alt | WARN | "IMG_2041.jpg" is not a description. |
| img-long-alt | WARN | Very long alt text cannot be navigated; move details to the text. |
| img-redundant-alt | INFO | "Image of" is redundant; screen readers already say "image". |
| img-decorative | INFO | `alt=""` hides the image; confirm it is truly decorative. |
See `alt-text-guide.md` for how to write the text.
## Tables
- table-empty-header (WARN): every column needs a header so screen readers can announce "Pro, Forecast days, 10". Add a header such as "Feature" to the first column.
- Use tables for data, not for layout. Keep them small; split very wide tables.
## Other rules
- color-only (WARN): "Press the green button" fails for color-blind readers and screen reader users. Name the button: "Select Save (the green button)".
- all-caps (INFO): long ALL CAPS text is harder to read, and some screen readers spell short caps words letter by letter. Use normal case and bold for emphasis.
- emoji-bullet (INFO): emoji are read aloud by name ("white heavy check mark Hourly forecasts"). Use normal list markers, and keep emoji rare and meaningful.
## Not checked by the script (check by hand)
- Whether alt text is accurate and useful.
- Color contrast and text inside images (screenshots with tiny text, charts with low contrast).
- Reading order of complex layouts, and HTML blocks embedded in Markdown.
- Captions and transcripts for videos and audio.
- Plain language: short sentences, defined jargon, and descriptive section names.
FILE:templates/a11y-review.md
# Accessibility Review: <document or folder>
Date: <YYYY-MM-DD>
Files checked: <n> (<list or folder>)
Audience: <who reads these docs>
Checker run: `python3 scripts/check_md_a11y.py <paths>` -> HIGH <n>, WARN <n>, INFO <n>
## Top fixes (most impact first)
1. <fix> (<file:line>)
2. <fix>
3. <fix>
## Alt text worksheet
| File:line | Image | Purpose in the document | Current alt | New alt | Seen the image? |
|---|---|---|---|---|---|
| | | informative / functional / decorative / complex | | | yes / draft |
## Headings
- Current outline: <H1 ... / H2 ... / H4 ...>
- Proposed outline: <H1 ... / H2 ... / H3 ...>
## Links
| File:line | Current text | New text | Destination |
|---|---|---|---|
| | | | |
## Other findings
| File:line | Rule | Change |
|---|---|---|
| | | |
## After fixes
Checker rerun: HIGH <n>, WARN <n>, INFO <n> (remaining INFO items explained below).
- <INFO item: reason it is acceptable>
## Manual checks still needed
- <for example: contrast of text inside screenshots, video captions>
FILE:examples/example-readme-review.md
# Example: accessibility review of a project README
## The request
"Our open source app's README gets a lot of traffic. A user who relies on a screen reader said it is hard to follow. Can you check it and fix it? The hero image is the app on a phone with the 3-day forecast; app-main.png is the main screen with hourly temperature, wind, and snow line; the badge is our build status."
## Checker run (before)
```
$ python3 scripts/check_md_a11y.py README.md
Markdown accessibility check: 1 file(s), 3 image(s)
HIGH 2 WARN 8 INFO 5
README.md (3 images, 4 headings): 15 finding(s)
3 HIGH img-missing-alt Image hero.png has empty alt text; describe it, or confirm it is purely decorative.
5 INFO all-caps Long ALL CAPS text is hard to read and some screen readers spell it letter by letter.
9 WARN vague-link Link text "here" does not say where it goes; describe the destination.
9 WARN vague-link Link text "click here" does not say where it goes; describe the destination.
11 WARN img-generic-alt Alt text "screenshot" for app-main.png is generic; say what the image shows or why it is there.
13 HIGH img-no-alt <img src="docs/img/badge.svg"> has no alt attribute.
15 WARN heading-skip Heading level jumps from H2 to H4 ("Configuration"); use H3.
17 WARN color-only Instruction relies on color; add the label, shape, or position too.
19 INFO fake-heading Bold line "Features" looks like a heading; use a real # heading if it starts a section.
21 INFO emoji-bullet Emoji used as a bullet is read aloud by name; use a list marker and put meaning in words.
22 INFO emoji-bullet Emoji used as a bullet is read aloud by name; use a list marker and put meaning in words.
24 WARN table-empty-header Table header has 1 empty cell(s); every column needs a header.
28 WARN same-text-different-url Link text "docs" points to 2 different URLs; make each text unique.
28 INFO bare-url Bare URL https://example.org/docs; screen readers spell it out. Use [descriptive text](https://example.org/docs).
35 WARN multiple-h1 2 H1 headings (lines 1, 35); keep one title and use ## for sections.
```
The `` inside the bash code block and the inline code example on the last line were correctly ignored.
## The review (filled template, shortened)
# Accessibility Review: README.md
Audience: hikers and contributors, many arriving from search.
Checker run: HIGH 2, WARN 8, INFO 5.
### Top fixes
1. Give the hero image and the build badge real alt text (lines 3 and 13).
2. Replace "here" and "click here" with link text that names the destination (line 9).
3. Fix the outline: "Configuration" becomes H3, "Features" becomes a real H3, and the second H1 "Contributing" becomes H2.
4. Name the Save button instead of only its color (line 17).
5. Give the first table column a header (line 24).
### Alt text worksheet
| File:line | Image | Purpose | Current alt | New alt | Seen? |
|---|---|---|---|---|---|
| README.md:3 | hero.png | informative: shows the product | (empty) | Trailhead Weather on a phone, showing a three-day forecast for a mountain pass. | described by the user |
| README.md:11 | app-main.png | informative: what the main screen shows | screenshot | Main screen with hourly temperature, wind, and snow line for the selected trailhead. | described by the user |
| README.md:13 | badge.svg | informative: build status | (no alt) | Build: passing | described by the user |
### Headings
- Before: H1 Trailhead Weather / H2 Install / H4 Configuration / H1 Contributing (plus a bold "Features" line).
- After: H1 Trailhead Weather / H2 Install / H3 Configuration / H3 Features / H2 Contributing.
### Links
| Line | Current text | New text | Destination |
|---|---|---|---|
| 9 | here | Download the latest release | https://example.org/releases |
| 9 | click here | setup guide | docs/guides/setup.md |
| 28 | docs (bare URL + "docs") | API reference | https://example.org/docs |
| 28 | docs | setup guide | docs/guides/setup.md |
### Other findings
| Line | Rule | Change |
|---|---|---|
| 5 | all-caps | Normal case inside the bold note. |
| 17 | color-only | "Select Save (the green button at the bottom of Settings)". |
| 21-22 | emoji-bullet | Normal "-" list items. |
| 24 | table-empty-header | First column header "Feature". |
### Corrected README (excerpt)
```markdown
# Trailhead Weather
Trailhead Weather shows mountain forecasts for hikers. **Note: this project is still in beta and things will break often.**
## Install
[Download the latest release](https://example.org/releases), or follow the [setup guide](docs/guides/setup.md).
<img src="docs/img/badge.svg" width="90" alt="Build: passing">
### Configuration
Select Save (the green button at the bottom of Settings) to save your settings.
### Features
- Hourly forecasts
- Avalanche bulletins
| Feature | Free | Pro |
|---|---|---|
| Forecast days | 3 | 10 |
```
### After fixes
```
$ python3 scripts/check_md_a11y.py README-fixed.md
Markdown accessibility check: 1 file(s), 3 image(s)
HIGH 0 WARN 0 INFO 0
```
### Manual checks still needed
- The hero image has small text on the phone screen; the alt text carries the key message, so that is fine here.
- If the README is ever turned into a docs site page, confirm the site does not add a second H1.
## Why this is a good answer
- Alt text describes each image's purpose, using what the user said the images show instead of guessing.
- Links and headings are fixed for how screen reader users navigate: by link list and by heading outline.
- The fix is verified by rerunning the checker, and the remaining manual checks are listed honestly.
FILE:scripts/check_md_a11y.py
#!/usr/bin/env python3
"""Check Markdown files for common accessibility problems.
Usage:
python3 check_md_a11y.py README.md docs/ # files and folders (*.md, *.markdown)
python3 check_md_a11y.py docs/ --json
cat page.md | python3 check_md_a11y.py -
python3 check_md_a11y.py docs/ --fail-on warn # exit 1 on WARN too (for CI)
Checks (code blocks and inline code are ignored):
images missing alt text, file-name or generic alt ("image", "screenshot"),
redundant "image of ..." openers, very long alt, HTML <img> without alt,
empty alt (fine only for decorative images)
headings no H1 or more than one H1, skipped levels (H2 -> H4), empty headings,
bold lines used as fake headings
links empty link text, vague text ("click here", "read more", "here"),
bare URLs, the same text pointing to different URLs
tables empty header cells
other ALL CAPS sentences, emoji used as bullets, color-only instructions
Severities: HIGH (blocks understanding), WARN (should fix), INFO (check by hand).
Exit code: 0 no HIGH, 1 HIGH found (or WARN with --fail-on warn), 2 usage error.
Standard library only.
"""
import argparse
import json
import os
import re
import sys
GENERIC_ALT = {"image", "img", "picture", "pic", "photo", "screenshot", "screen shot", "graphic", "icon",
"logo", "diagram", "chart", "figure", "untitled", "alt", "alt text", "banner", "placeholder"}
VAGUE_LINK = {"click here", "here", "click", "read more", "more", "learn more", "link", "this", "this link",
"this page", "go", "details", "more info", "info", "see here", "check it out"}
REDUNDANT_START = re.compile(r"^(an? )?(image|picture|photo|screenshot|graphic) (of|showing)\b", re.I)
FILE_ALT = re.compile(r"^[\w\-. ]+\.(png|jpe?g|gif|svg|webp|bmp|tiff?)$|^(img|image|dsc|screenshot)[_\- ]?\d+", re.I)
IMG_MD = re.compile(r"!\[([^\]]*)\]\(\s*<?([^)\s>]*)>?(?:\s+\"[^\"]*\")?\s*\)")
IMG_REF = re.compile(r"!\[([^\]]*)\]\[[^\]]*\]")
LINK_MD = re.compile(r"(?<!!)\[([^\]]*)\]\(\s*<?([^)\s>]*)>?(?:\s+\"[^\"]*\")?\s*\)")
HTML_IMG = re.compile(r"<img\b[^>]*>", re.I)
BARE_URL = re.compile(r"(?<![(<\"'=\[])\bhttps?://[^\s)>\]]+")
COLOR_ONLY = re.compile(r"\b(click|press|tap|select|choose|use)\b[^.]{0,30}\b(red|green|blue|yellow|orange|purple)\b(?! *\()", re.I)
EMOJI_BULLET = re.compile(r"^\s*[\u2705\u274c\u2b50\u27a1\u2714\u2716\U0001F300-\U0001FAFF]\ufe0f?\s+\S")
def strip_inline_code(line):
return re.sub(r"`+[^`]*`+", lambda m: " " * len(m.group(0)), line)
def check_text(text, name):
findings = []
def add(sev, line, rule, msg):
findings.append({"file": name, "line": line, "severity": sev, "rule": rule, "message": msg})
lines = text.replace("\r\n", "\n").split("\n")
in_fence, fence = False, ""
headings, link_targets = [], {}
images = 0
prev_blank = True
for no, raw in enumerate(lines, 1):
stripped = raw.strip()
m = re.match(r"^\s{0,3}(`{3,}|~{3,})", raw)
if m:
if not in_fence:
in_fence, fence = True, m.group(1)[0] * 3
elif stripped.startswith(fence):
in_fence = False
continue
if in_fence or raw.startswith(" ") and prev_blank and not stripped.startswith(("-", "*", "+")):
prev_blank = not stripped if not in_fence else prev_blank
continue
line = strip_inline_code(raw)
hm = re.match(r"^\s{0,3}(#{1,6})(\s+|$)(.*?)\s*#*\s*$", line)
if hm:
level, title = len(hm.group(1)), hm.group(3).strip()
headings.append((no, level, title))
if not title:
add("HIGH", no, "empty-heading", f"Empty H{level} heading; screen reader users navigate by headings.")
elif re.fullmatch(r"\s*(\*\*|__)[^*_]{2,80}(\*\*|__):?\s*", line) and prev_blank:
add("INFO", no, "fake-heading", f"Bold line \"{stripped.strip('*_:')}\" looks like a heading; use a real # heading if it starts a section.")
images += len(IMG_MD.findall(line)) + len(IMG_REF.findall(line)) + len(HTML_IMG.findall(line))
for alt, src in IMG_MD.findall(line):
check_alt(add, no, alt, src)
for alt in IMG_REF.findall(line):
check_alt(add, no, alt, "(reference)")
for tag in HTML_IMG.findall(line):
am = re.search(r"\balt\s*=\s*(\"([^\"]*)\"|'([^']*)'|(\S+))", tag, re.I)
src = re.search(r"\bsrc\s*=\s*[\"']?([^\"'\s>]+)", tag, re.I)
src = src.group(1) if src else "?"
if not am:
add("HIGH", no, "img-no-alt", f"<img src=\"{src}\"> has no alt attribute.")
else:
check_alt(add, no, am.group(2) if am.group(2) is not None else (am.group(3) if am.group(3) is not None else am.group(4)), src, html=True)
no_images = IMG_MD.sub(" ", line)
for text_, url in LINK_MD.findall(no_images):
label = re.sub(r"[*_`]", "", text_).strip()
if not label:
add("HIGH", no, "empty-link", f"Link to {url or '(no url)'} has no text; screen readers will read the URL or nothing.")
elif label.lower().rstrip(".!:") in VAGUE_LINK:
add("WARN", no, "vague-link", f"Link text \"{label}\" does not say where it goes; describe the destination.")
elif re.fullmatch(r"https?://\S+", label):
add("INFO", no, "url-as-link-text", f"Link text is a raw URL ({label[:60]}); use a short description.")
if label:
link_targets.setdefault(label.lower(), {}).setdefault(url, no)
for url in BARE_URL.findall(LINK_MD.sub(" ", no_images)):
add("INFO", no, "bare-url", f"Bare URL {url[:60]}; screen readers spell it out. Use [descriptive text]({url[:40]}).")
if "|" in line and re.match(r"^\s*\|?\s*:?-{3,}:?\s*(\|\s*:?-{3,}:?\s*)+\|?\s*$", line) and no > 1:
header = lines[no - 2].strip().strip("|")
cells = [c.strip() for c in header.split("|")]
if any(not c for c in cells):
add("WARN", no - 1, "table-empty-header", f"Table header has {sum(1 for c in cells if not c)} empty cell(s); every column needs a header.")
run = longest = 0
for w in re.findall(r"[A-Za-z]+", line):
run = run + 1 if w.isupper() and (len(w) > 1 or w in "AI") else 0
longest = max(longest, run)
if longest >= 5 and not hm:
add("INFO", no, "all-caps", "Long ALL CAPS text is hard to read and some screen readers spell it letter by letter.")
if EMOJI_BULLET.match(line):
add("INFO", no, "emoji-bullet", "Emoji used as a bullet is read aloud by name; use a list marker and put meaning in words.")
cm = COLOR_ONLY.search(line)
if cm:
between = line[cm.end(1):cm.start(2)]
inside_parens = line[:cm.start(2)].count("(") > line[:cm.start(2)].count(")")
names_label = re.match(r"\s+(\*\*|\"|')?[A-Z]", between) # "Select Save ..." or "Press **Save**"
if inside_parens or names_label:
cm = None
if cm:
add("WARN", no, "color-only", "Instruction relies on color; add the label, shape, or position too.")
prev_blank = not stripped
h1 = [h for h in headings if h[1] == 1]
if headings and not h1:
add("WARN", headings[0][0], "no-h1", "Document has no H1; start with one # title.")
if len(h1) > 1:
add("WARN", h1[1][0], "multiple-h1", f"{len(h1)} H1 headings (lines {', '.join(str(h[0]) for h in h1)}); keep one title and use ## for sections.")
for (pno, plev, _), (no, lev, title) in zip(headings, headings[1:]):
if lev > plev + 1:
add("WARN", no, "heading-skip", f"Heading level jumps from H{plev} to H{lev} (\"{title[:50]}\"); use H{plev + 1}.")
if not headings and len([l for l in lines if l.strip()]) > 15:
add("INFO", 1, "no-headings", "Long document with no headings; add headings so readers can navigate.")
for label, targets in link_targets.items():
if len(targets) > 1 and label not in VAGUE_LINK:
first = min(targets.values())
add("WARN", first, "same-text-different-url", f"Link text \"{label}\" points to {len(targets)} different URLs; make each text unique.")
order = {"HIGH": 0, "WARN": 1, "INFO": 2}
findings.sort(key=lambda f: (f["line"], order[f["severity"]]))
return findings, len(headings), images
def check_alt(add, no, alt, src, html=False):
a = alt.strip()
base = os.path.basename(src) if src else ""
if a == "" and html:
add("INFO", no, "img-decorative", f"<img src=\"{base}\"> has alt=\"\" (marked decorative); confirm it carries no information.")
return
if a == "" and src:
add("HIGH", no, "img-missing-alt", f"Image {base or src} has empty alt text; describe it, or confirm it is purely decorative.")
return
low = a.lower().rstrip(".")
if low in GENERIC_ALT:
add("WARN", no, "img-generic-alt", f"Alt text \"{a}\" for {base} is generic; say what the image shows or why it is there.")
elif FILE_ALT.search(a) or (base and low == os.path.splitext(base)[0].lower()):
add("WARN", no, "img-filename-alt", f"Alt text \"{a}\" looks like a file name; describe the image instead.")
elif REDUNDANT_START.match(a):
add("INFO", no, "img-redundant-alt", f"Alt text starts with \"{' '.join(a.split()[:3])}\"; screen readers already say 'image', so start with the content.")
if len(a) > 150:
add("WARN", no, "img-long-alt", f"Alt text is {len(a)} characters; keep it under about 150 and move details to the surrounding text or a caption.")
def collect(paths):
files = []
for p in paths:
if p == "-":
files.append("-")
elif os.path.isdir(p):
for root, dirs, names in os.walk(p):
dirs[:] = sorted(d for d in dirs if not d.startswith(".") and d != "node_modules")
files += [os.path.join(root, n) for n in sorted(names) if n.lower().endswith((".md", ".markdown"))]
elif os.path.isfile(p):
files.append(p)
else:
raise FileNotFoundError(p)
return files
def main():
ap = argparse.ArgumentParser(description="Markdown accessibility checker")
ap.add_argument("paths", nargs="+", help="Markdown files, folders, or - for stdin")
ap.add_argument("--json", action="store_true")
ap.add_argument("--fail-on", choices=["high", "warn"], default="high")
args = ap.parse_args()
try:
files = collect(args.paths)
except FileNotFoundError as e:
print(f"error: path not found: {e}", file=sys.stderr)
return 2
if not files:
print("error: no Markdown files (*.md, *.markdown) found", file=sys.stderr)
return 2
all_findings, stats = [], []
for f in files:
text = sys.stdin.read() if f == "-" else open(f, encoding="utf-8", errors="replace").read()
name = "<stdin>" if f == "-" else f
found, nhead, images = check_text(text, name)
stats.append({"file": name, "images": images, "headings": nhead, "findings": len(found)})
all_findings += found
counts = {s: sum(1 for x in all_findings if x["severity"] == s) for s in ("HIGH", "WARN", "INFO")}
if args.json:
print(json.dumps({"files": stats, "counts": counts, "findings": all_findings}, indent=2))
else:
print(f"Markdown accessibility check: {len(files)} file(s), {sum(s['images'] for s in stats)} image(s)")
print(f"HIGH {counts['HIGH']} WARN {counts['WARN']} INFO {counts['INFO']}")
for s in stats:
items = [x for x in all_findings if x["file"] == s["file"]]
print(f"\n{s['file']} ({s['images']} images, {s['headings']} headings): {len(items)} finding(s)")
for x in items:
print(f" {x['line']:>4} {x['severity']:<4} {x['rule']:<24} {x['message']}")
print("\nAutomated checks cover only part of accessibility; review alt text meaning, reading order, and contrast by hand.")
bad = counts["HIGH"] + (counts["WARN"] if args.fail_on == "warn" else 0)
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())【中文说明】用于把零散会议议题整理为带起止时间、负责人和明确产出的限时议程，并检查缓冲、休息及决策安排。发送提示词后提供会议时段、议题、参与者与优先级，即可请求可直接发送的议程和会前阅读材料。 Turns a list of meeting topics into a timed agenda with clock times, owners, and an outcome per item, checks it fits the slot with a buffer, flags late or rushed decisions, too many status updates, oversized items, and missing breaks, and writes the agenda to send. Includes a tested stdlib Python timeboxer with a fit mode.
---
name: meeting-agenda-timeboxer
description: Turns a rough list of meeting topics into a timeboxed agenda with start and end times, owners, and a clear outcome for every item (inform, discuss, decide, brainstorm, review), checks that it fits the slot with a buffer, flags agenda smells such as late or rushed decisions, too many status updates, oversized items, and missing breaks, and writes a ready-to-send agenda and pre-read. Use when a user asks "help me plan this meeting", "will this agenda fit in an hour?", "make this agenda tighter", or shares a list of topics for a team meeting, workshop, offsite, or review.
---
# Meeting Agenda Timeboxer
You help people run meetings that end on time and produce decisions. You turn a list of topics into a realistic schedule, give every item an owner and a purpose, move status updates out of the room, and make the time math visible so nobody discovers at minute 55 that the main decision has no time left.
## Files in this skill
- `scripts/timebox_agenda.py` - schedules every item, checks the fit against the slot and buffer, flags agenda smells, and can fit items to the slot in 5-minute steps (Python 3 standard library only)
- `references/timeboxing-principles.md` - how long things really take, item outcomes, ordering, buffers, and breaks
- `references/facilitation-moves.md` - techniques to keep each kind of item inside its timebox, and what to do when one runs over
- `templates/agenda.md` - the agenda input format and the agenda to send
- `examples/example-roadmap-meeting.md` - a worked fix of an overloaded one-hour planning meeting
## Workflow
### 1. Collect the facts
Ask, or assume and say so:
- Purpose of the meeting in one sentence: what must be true when it ends?
- Start time and length (or end time), number of attendees, in person or remote.
- Topics, each with an owner and what the group must do with it: inform, update, discuss, brainstorm, review, or decide.
- Hard constraints: people who join late or leave early, fixed breaks.
### 2. Write the draft agenda
Use the input format in `templates/agenda.md` (one line per item: `minutes | topic | owner | outcome`). Use `?` for a duration nobody knows yet. Estimate durations with `references/timeboxing-principles.md`.
### 3. Run the timeboxer
```bash
python3 scripts/timebox_agenda.py agenda.txt
python3 scripts/timebox_agenda.py agenda.txt --fit # scale to the slot in 5-minute steps
python3 scripts/timebox_agenda.py agenda.json --json
```
It prints the schedule with clock times, the buffer, and findings: HIGH when the agenda does not fit, WARN for smells, INFO for unplanned time and the meeting cost in person-hours. `--fit` trims status items first, then scales everything else; it never reorders items, so check the order yourself afterwards.
If you cannot run the script, do the clock math by hand, item by item, and say so.
### 4. Fix the agenda, not just the numbers
Work through the findings in this order:
1. Remove or shrink status updates and announcements: move them into a written pre-read.
2. Put decisions early (first half), and give each one at least 10 minutes plus a written proposal sent beforehand.
3. Give every item an owner and an outcome; drop "AOB" or turn it into a 5-minute parking lot review.
4. Split items over 45 minutes; add a break every 60 to 90 minutes.
5. If it still does not fit, propose what to cut or move to another meeting or to async, and say what is lost.
Rerun the script until there are no HIGH findings and every WARN is either fixed or explained.
### 5. Deliver
Fill in the agenda to send from `templates/agenda.md`: purpose, desired outcomes, the timed schedule, pre-reads with owners and deadlines, and the parking lot rule. Add one or two facilitation tips per risky item from `references/facilitation-moves.md`. Follow the style of `examples/example-roadmap-meeting.md`.
## Rules
- Never make the math work by shrinking a decision below what it needs; cut or move something else, or recommend a longer slot.
- Keep a buffer: about 5 minutes per hour, 10 percent for workshops.
- Say plainly when a meeting should not happen (no decisions, nothing to discuss) and suggest the async alternative.
- Respect the user's culture and constraints; suggest, do not lecture.
- Times are local to the meeting; for remote attendees in other time zones, mention it but do not convert unless asked.
FILE:references/timeboxing-principles.md
# Timeboxing principles
## Every item needs an outcome
| Outcome | What the group does | Typical time | Notes |
|---|---|---|---|
| inform | Hear something, ask a clarifying question | 2 to 5 min | Better as a written pre-read. |
| update | Status from people or teams | 1 to 2 min per person | Usually async; keep only blockers live. |
| discuss | Explore a topic, share views, no decision yet | 10 to 20 min | Say what the discussion feeds (a decision next week, a doc). |
| brainstorm | Generate options | 15 to 30 min | Needs a framed question and a method (silent writing first). |
| review | Give feedback on a draft, design, or plan | 15 to 40 min | Send the material beforehand; agree what kind of feedback is wanted. |
| decide | Choose between options and commit | 10 to 30 min | Needs a written proposal, the options, the decision maker, and the decision rule. |
| break | Rest | 5 to 15 min | Every 60 to 90 minutes. |
If an item has no clear outcome, ask: "What will be different after this item?" If the answer is "nothing", cut it.
## How long things really take
- Introductions: about 1 minute per person for a new group, 0 for a regular team.
- Each person speaking once on a topic: about 1.5 to 2 minutes per person. Eight people, one round, is 15 minutes.
- Screen sharing and getting started: 2 to 3 minutes, more for remote meetings with new tools.
- A decision with real disagreement: double the estimate, or split it into "discuss" now and "decide" next time.
- Items late in the agenda run faster but worse: people rush or leave.
## Ordering
1. Start with a short framing: purpose and desired outcomes (2 to 5 min).
2. Decisions and the hardest topics in the first half, while energy and attendance are high.
3. Discussion and review in the middle.
4. Information and updates last or out of the meeting entirely.
5. End with next steps, owners, and dates (5 min), never with AOB.
Group items that need the same people, so others can leave early.
## Buffers and breaks
- Buffer: about 5 minutes per hour, 10 percent for workshops. The buffer is part of the slot, not extra time.
- Breaks: 5 to 10 minutes every 60 to 90 minutes; 15 for longer workshops. Remote meetings need breaks sooner.
- Back-to-back calendars: end 5 minutes early (25 or 50 minute meetings).
## Fit checks
- More than about 8 items per hour means each gets less than 7 minutes; merge or cut.
- Status and announcements above about a third of the time: move them to a pre-read.
- An item over 45 minutes: split it into steps with their own outcomes (for example "brainstorm 25" then "cluster and pick 20").
## The cost of a meeting
Person-hours = attendees x length in hours. A one-hour meeting with 9 people costs 9 working hours. Ask whether everyone needs to be there for every item.
FILE:references/facilitation-moves.md
# Facilitation moves that keep timeboxes
## Before the meeting
- Send the agenda and pre-reads 24 to 48 hours ahead, with the owner of each item.
- For every decision item, send a one-page proposal: the question, options, recommendation, and who decides.
- Ask people to add questions to the pre-read in writing, so the meeting starts from questions, not from a presentation.
## At the start
- State the purpose and the outcomes you need by the end.
- Name a timekeeper and agree on the signal (a message in the chat, a raised hand at 2 minutes left).
- Say how decisions are made today: the owner decides after input, majority vote, or consent ("any strong objection?").
## By item type
- Inform: "Two minutes, then one clarifying question each." Detailed questions go to the parking lot or to a follow-up chat.
- Update: Only blockers and asks live; everything else is in the pre-read.
- Discuss: Frame the question first ("What worries us about option B?"). Use a quick round so everyone speaks once before anyone speaks twice.
- Brainstorm: 3 to 5 minutes of silent writing first, then share, then cluster. Quantity before judgment.
- Review: Agree the kind of feedback: blocking issues, questions, or polish. Collect blocking issues first.
- Decide: Restate the proposal, list concerns, check for objections, then decide and say it out loud: "We decided X; owner Y; by date Z."
## When an item runs over
Make the choice visible instead of silently eating the next item:
1. "We have 2 minutes left on this. Do we extend by 5 minutes and cut something else, or park it?"
2. Park it: write it in the parking lot with an owner and a date.
3. Shrink the next item, never the decision items.
4. If the meeting goal is at risk, schedule a short follow-up with only the people needed.
## Remote and hybrid
- Everyone joins on their own device so remote people are equal.
- Use the chat for the speaking queue and for quick polls.
- Share the agenda on screen with the current item highlighted.
## At the end
- Read back decisions, owners, and dates (5 minutes).
- Review the parking lot: assign each item or drop it.
- Send notes within 24 hours.
FILE:templates/agenda.md
# Agenda input (for scripts/timebox_agenda.py)
```
Meeting: <name>
Start: <HH:MM>
Length: <minutes> # or End: <HH:MM>
Buffer: <minutes or percent, e.g. 5 or 10%>
Attendees: <number>
<minutes> | <topic> | <owner> | <inform | update | discuss | brainstorm | review | decide | break>
? | <topic with unknown duration> | <owner> | <outcome>
```
A Markdown table with the columns Min, Topic, Owner, Outcome works too, and so does JSON:
`{"meeting": "...", "start": "14:00", "length": 60, "buffer": 5, "attendees": 6, "items": [{"minutes": 10, "topic": "...", "owner": "...", "outcome": "decide"}]}`
---
# Agenda to send
**<Meeting name>**
<Day, date>, <start>-<end> (<length> min), <place or video link>
Attendees: <names or teams> | Facilitator: <name> | Timekeeper: <name> | Notes: <name>
**Purpose:** <one sentence: what must be true when we finish>
**Outcomes we need**
1. <decision or result>
2. <decision or result>
**Pre-read (by <day, time>)**
- <document> - <owner> - <what to look at, how long it takes to read>
**Schedule**
| Time | Min | Item | Owner | Outcome |
|---|---|---|---|---|
| <HH:MM-HH:MM> | <n> | <topic> | <owner> | <outcome> |
| <HH:MM-HH:MM> | <n> | Buffer and wrap-up: decisions, owners, dates | <facilitator> | |
**How we decide today:** <owner decides after input | consent | vote>
**Parking lot:** Off-topic points go to the parking lot and get an owner at the end.
**Moved out of this meeting**
- <item> -> <async update, other meeting, or a document>, owner <name>
FILE:examples/example-roadmap-meeting.md
# Example: fixing an overloaded one-hour planning meeting
## The request
"I have one hour with 9 people on Thursday at 14:00 to plan Q4. Here is my agenda. Will it fit? The budget split must be approved in this meeting, and the top 3 initiatives too."
Draft agenda (`agenda.txt`):
```
Meeting: Q4 roadmap planning
Start: 14:00
Length: 60
Buffer: 5
Attendees: 9
10 | Welcome and goals | Dana | inform
15 | Team status round (each team lead) | Leads | update
10 | Customer feedback highlights | Priya | inform
20 | Q4 priorities: pick top 3 initiatives | Lee | discuss
10 | Hiring plan for Q4 | | discuss
5 | Approve Q4 budget split | Dana | decide
? | AOB | |
```
## First run
```
$ python3 scripts/timebox_agenda.py agenda.txt
Q4 roadmap planning: 14:00-15:00 (60 min, buffer 5 min, 55 min for items)
Planned: 70 min
14:00-14:10 10 min Welcome and goals (Dana) [inform]
14:10-14:25 15 min Team status round (each team lead) (Leads) [update]
14:25-14:35 10 min Customer feedback highlights (Priya) [inform]
14:35-14:55 20 min Q4 priorities: pick top 3 initiatives (Lee) [discuss]
14:55-15:05 10 min Hiring plan for Q4 [discuss]
15:05-15:10 5 min Approve Q4 budget split (Dana) [decide]
15:10-15:10 ? min AOB [?]
14:55-15:00 5 min Buffer / wrap-up
Findings:
- [HIGH] Agenda needs 70 min but only 55 min are available after the 5-min buffer: overrun of 15 min (ends 15:10 instead of 14:55). Cut, shorten, or move items, or run with --fit.
- [WARN] 1 item(s) without a duration: AOB. Remaining time for them: 0 min.
- [WARN] "Hiring plan for Q4" has no owner; name who prepares and leads it.
- [WARN] "Approve Q4 budget split" is a decision with only 5 min; plan at least 10, or decide by a written proposal beforehand.
- [WARN] "Approve Q4 budget split" is a decision starting at 15:05, in the last quarter of the meeting; move it earlier while energy and attendance are high.
- [WARN] "AOB" has no owner; name who prepares and leads it.
- [WARN] "AOB" has no outcome; say whether it is to inform, discuss, decide, or brainstorm.
- [WARN] Status updates and announcements take 50% of the time; send them as a pre-read and keep the meeting for discussion and decisions.
- [INFO] Meeting cost: 9 people x 60 min = 9 person-hours.
```
Exit code 1: the agenda does not fit.
## What the numbers say
- 70 minutes of items (plus AOB) in a 55-minute window: 15 minutes over before AOB even starts.
- Half of the time is updates and announcements, while the two things that must be decided get 5 minutes (budget) and a "discuss" label (priorities), and the budget decision would start at 15:05, after the meeting should have ended.
- Running `--fit` alone would squeeze everything into 55 minutes but leave the budget decision last with 5 minutes. Fitting the numbers is not enough; the order and the purpose of items have to change.
## Changes
1. Team status round (15 min) and customer feedback highlights (10 min) become a written pre-read due Wednesday 12:00; questions go in comments.
2. The budget split moves to the start with 10 minutes and a proposal sent Monday, so the meeting only confirms or adjusts it.
3. "Q4 priorities" becomes a decision with 25 minutes: the outcome is a ranked top 3.
4. The hiring plan gets an owner (Priya) and a narrower question: which roles to open.
5. AOB is replaced by a 5-minute "next steps, owners, and parking lot" close.
6. Three people only needed for the status round can skip the meeting: 6 attendees instead of 9.
Revised agenda:
```
Meeting: Q4 roadmap planning
Start: 14:00
Length: 60
Buffer: 5
Attendees: 6
5 | Welcome and goals for today | Dana | inform
10 | Approve Q4 budget split (proposal sent Monday) | Dana | decide
25 | Q4 priorities: pick top 3 initiatives | Lee | decide
10 | Hiring plan for Q4: agree which roles to open | Priya | discuss
5 | Next steps, owners, and parking lot | Dana | inform
```
## Second run
```
$ python3 scripts/timebox_agenda.py agenda-revised.txt
Q4 roadmap planning: 14:00-15:00 (60 min, buffer 5 min, 55 min for items)
Planned: 55 min
14:00-14:05 5 min Welcome and goals for today (Dana) [inform]
14:05-14:15 10 min Approve Q4 budget split (proposal sent Monday) (Dana) [decide]
14:15-14:40 25 min Q4 priorities: pick top 3 initiatives (Lee) [decide]
14:40-14:50 10 min Hiring plan for Q4: agree which roles to open (Priya) [discuss]
14:50-14:55 5 min Next steps, owners, and parking lot (Dana) [inform]
14:55-15:00 5 min Buffer / wrap-up
Findings:
- [INFO] Meeting cost: 6 people x 60 min = 6 person-hours.
```
Exit code 0, no warnings, and the meeting costs 6 person-hours instead of 9.
## The agenda to send (filled template, shortened)
**Q4 roadmap planning**
Thursday, 14:00-15:00 (60 min), Room 3 and video link
Facilitator: Dana | Timekeeper: Lee | Notes: Priya
**Purpose:** Leave with an approved Q4 budget split and a ranked top 3 of initiatives.
**Pre-read (by Wednesday 12:00)**
- Team status one-pagers - each team lead - 5 min to read
- Customer feedback highlights - Priya - 5 min
- Budget split proposal - Dana - 10 min
**Schedule**
| Time | Min | Item | Owner | Outcome |
|---|---|---|---|---|
| 14:00-14:05 | 5 | Welcome and goals for today | Dana | inform |
| 14:05-14:15 | 10 | Approve Q4 budget split (proposal sent Monday) | Dana | decide |
| 14:15-14:40 | 25 | Q4 priorities: pick top 3 initiatives | Lee | decide |
| 14:40-14:50 | 10 | Hiring plan for Q4: agree which roles to open | Priya | discuss |
| 14:50-14:55 | 5 | Next steps, owners, and parking lot | Dana | inform |
| 14:55-15:00 | 5 | Buffer | | |
**How we decide today:** Dana decides the budget after input; the top 3 is chosen by consent (any strong objection?).
**Facilitation tips**
- Priorities: 3 minutes of silent dot voting on the candidate list, then discuss only the items ranked 2 to 5.
- If the budget item hits 10 minutes, park the open point with an owner and a 24-hour deadline instead of eating into priorities.
**Moved out of this meeting:** team status round and customer feedback (pre-read).
## Why this is a good answer
- It fixes the purpose and the order, not just the minutes.
- Both required decisions now happen in the first 40 minutes, with prepared proposals.
- The script confirms the fit, and the people who were only there for updates get their hour back.
FILE:scripts/timebox_agenda.py
#!/usr/bin/env python3
"""Timebox a meeting agenda: schedule every item, check the fit, and flag agenda smells.
Usage:
python3 timebox_agenda.py agenda.txt
python3 timebox_agenda.py agenda.txt --fit # scale items to fit the slot (5-minute steps)
python3 timebox_agenda.py agenda.json --json
cat agenda.txt | python3 timebox_agenda.py -
Text input: optional "Key: value" header lines, then one item per line as
minutes | topic | owner | outcome
Markdown table rows (| 10 | Topic | Owner | decide |) also work; header and
separator rows are skipped. Use ? for an unknown duration.
Header keys: Meeting, Start (HH:MM), Length (minutes) or End (HH:MM),
Buffer (minutes or percent, e.g. 5 or 10%), Attendees (number).
Outcomes: inform, update, discuss, brainstorm, review, decide, break.
JSON input: {"meeting": "...", "start": "14:00", "length": 60, "buffer": 5,
"attendees": 8, "items": [{"minutes": 10, "topic": "...", "owner": "...", "outcome": "decide"}]}
--fit trims status items (and items with no outcome) first, then scales everything; spare time goes to
decide and discuss items.
Checks: overrun (HIGH), missing durations, owners, or outcomes, late or rushed
decisions, oversized items, long stretches without a break, too many status
updates, too many items per hour, plus the meeting's cost in person-hours.
Exit code: 0 no HIGH findings, 1 HIGH found, 2 usage or input error.
Standard library only.
"""
import argparse
import json
import math
import re
import sys
OUTCOMES = {"inform", "update", "discuss", "brainstorm", "review", "decide", "break"}
ALIASES = {"decision": "decide", "info": "inform", "fyi": "inform", "status": "update", "updates": "update",
"discussion": "discuss", "ideas": "brainstorm", "ideation": "brainstorm", "feedback": "review",
"approve": "decide", "approval": "decide", "pause": "break", "coffee": "break"}
class InputError(Exception):
pass
def hhmm(text):
m = re.fullmatch(r"\s*(\d{1,2})[:.](\d{2})\s*", str(text))
if not m or int(m.group(1)) > 23 or int(m.group(2)) > 59:
raise InputError(f"time must be HH:MM, got {text!r}")
return int(m.group(1)) * 60 + int(m.group(2))
def fmt(minutes):
minutes = int(round(minutes)) % (24 * 60)
return f"{minutes // 60:02d}:{minutes % 60:02d}"
def parse_minutes(text):
t = str(text).strip().lower()
if t in ("", "?", "tbd", "-"):
return None
m = re.fullmatch(r"(\d+(?:\.\d+)?)\s*(m|min|mins|minutes?)?", t)
if m:
return float(m.group(1))
m = re.fullmatch(r"(\d+(?:\.\d+)?)\s*h(?:ours?)?(?:\s*(\d+)\s*m(?:in)?)?", t)
if m:
return float(m.group(1)) * 60 + float(m.group(2) or 0)
raise InputError(f"cannot read duration {text!r}")
def norm_outcome(text):
t = (text or "").strip().lower()
return ALIASES.get(t, t)
def parse_text(text):
meta, items, errors = {}, [], []
for no, raw in enumerate(text.replace("\r\n", "\n").split("\n"), 1):
line = raw.strip()
if not line or line.startswith("#"):
continue
hm = re.fullmatch(r"([A-Za-z ]{3,20}):\s*(.+)", line)
if hm and "|" not in line:
meta[hm.group(1).strip().lower()] = hm.group(2).strip()
continue
if "|" not in line:
errors.append(f"line {no}: expected 'minutes | topic | owner | outcome', got {line[:60]!r}")
continue
cells = [c.strip() for c in line.strip("|").split("|")]
if all(re.fullmatch(r":?-{2,}:?", c) for c in cells if c):
continue
if cells and cells[0].lower() in ("min", "mins", "minutes", "time", "duration"):
continue
cells += [""] * (4 - len(cells))
try:
minutes = parse_minutes(cells[0])
except InputError as e:
errors.append(f"line {no}: {e}")
continue
if not cells[1]:
errors.append(f"line {no}: item has no topic")
continue
items.append({"line": no, "minutes": minutes, "topic": cells[1], "owner": cells[2],
"outcome": norm_outcome(cells[3])})
return meta, items, errors
def parse_json(text):
data = json.loads(text)
if not isinstance(data, dict) or not isinstance(data.get("items"), list):
raise InputError("JSON input needs an object with an 'items' list")
meta = {k.lower(): str(v) for k, v in data.items() if k != "items" and v is not None}
items, errors = [], []
for i, it in enumerate(data["items"], 1):
try:
minutes = parse_minutes(it.get("minutes", ""))
except InputError as e:
errors.append(f"item {i}: {e}")
continue
if not str(it.get("topic", "")).strip():
errors.append(f"item {i}: item has no topic")
continue
items.append({"line": i, "minutes": minutes, "topic": str(it["topic"]).strip(),
"owner": str(it.get("owner", "") or "").strip(), "outcome": norm_outcome(it.get("outcome", ""))})
return meta, items, errors
def settings(meta):
start = hhmm(meta.get("start", "09:00"))
if "length" in meta:
length = parse_minutes(meta["length"])
if not length or length <= 0:
raise InputError("Length must be a positive number of minutes")
elif "end" in meta:
length = (hhmm(meta["end"]) - start) % (24 * 60)
if length == 0:
raise InputError("End must be after Start")
else:
raise InputError("header needs 'Length: <minutes>' or 'End: HH:MM'")
b = meta.get("buffer", "0").strip()
if b.endswith("%"):
buffer = 5 * round(length * float(b[:-1]) / 100 / 5) # percent buffers round to 5 minutes
else:
buffer = parse_minutes(b) or 0
if buffer >= length:
raise InputError("Buffer must be shorter than the meeting")
attendees = int(meta["attendees"]) if str(meta.get("attendees", "")).strip().isdigit() else None
return {"meeting": meta.get("meeting", meta.get("title", "Meeting")), "start": start, "length": length,
"buffer": buffer, "available": length - buffer, "attendees": attendees}
def fit(items, available):
"""Fit items into `available` minutes in 5-minute steps (missing durations count as 10, breaks kept).
When the agenda is too long, status items (inform, update, or no outcome) are trimmed first, down to half
their planned time; then every item is scaled proportionally. When it is too short, the
spare time goes to decide and discuss items first.
"""
fixed = sum(it["minutes"] or 0 for it in items if it["outcome"] == "break")
flex = [it for it in items if it["outcome"] != "break"]
room = available - fixed
if not flex or room < 5 * len(flex):
raise InputError(f"cannot fit {len(flex)} items into {room:g} minutes with at least 5 minutes each")
want = [max(5, 5 * round((it["minutes"] or 10) / 5)) for it in flex]
cur = list(want)
status = [i for i, it in enumerate(flex) if it["outcome"] in ("inform", "update", "")]
while sum(cur) > room:
cands = [i for i in status if cur[i] - 5 >= max(5, want[i] / 2)]
if not cands:
break
cur[max(cands, key=lambda k: cur[k])] -= 5
if sum(cur) > room:
scale = room / sum(cur)
ref = [c * scale for c in cur]
cur = [max(5, 5 * round(r / 5)) for r in ref]
while sum(cur) > room:
cands = [k for k in range(len(cur)) if cur[k] > 5]
cur[max(cands, key=lambda k: (cur[k] - ref[k], cur[k]))] -= 5
while sum(cur) + 5 <= room:
cur[min(range(len(cur)), key=lambda k: cur[k] - ref[k])] += 5
else:
prio = [i for i, it in enumerate(flex) if it["outcome"] in ("decide", "discuss")] or list(range(len(flex)))
while sum(cur) + 5 <= room:
cur[min(prio, key=lambda k: cur[k] / want[k])] += 5
out, it_new = [], iter(cur)
for it in items:
copy = dict(it)
copy["original"] = it["minutes"]
if it["outcome"] != "break":
copy["minutes"] = float(next(it_new))
out.append(copy)
return out
def analyze(items, cfg):
findings = []
def add(sev, msg):
findings.append({"severity": sev, "message": msg})
t = cfg["start"]
for it in items:
it["start"] = t
t += it["minutes"] or 0
it["end"] = t
planned = sum(it["minutes"] or 0 for it in items)
avail, end_slot = cfg["available"], cfg["start"] + cfg["length"]
unknown = [it for it in items if it["minutes"] is None]
if planned > avail:
add("HIGH", f"Agenda needs {planned:g} min but only {avail:g} min are available"
f"{' after the ' + format(cfg['buffer'], 'g') + '-min buffer' if cfg['buffer'] else ''}: overrun of {planned - avail:g} min"
f" (ends {fmt(cfg['start'] + planned)} instead of {fmt(end_slot - cfg['buffer'])}). Cut, shorten, or move items, or run with --fit.")
elif avail - planned > 0.15 * avail and not unknown:
add("INFO", f"{avail - planned:g} min unplanned; end early or give the main decision more time.")
if unknown:
add("WARN", f"{len(unknown)} item(s) without a duration: " + "; ".join(it["topic"][:40] for it in unknown)
+ f". Remaining time for them: {max(avail - planned, 0):g} min.")
for it in items:
name = f"\"{it['topic'][:50]}\""
if it["outcome"] != "break" and not it["owner"]:
add("WARN", f"{name} has no owner; name who prepares and leads it.")
if not it["outcome"]:
add("WARN", f"{name} has no outcome; say whether it is to inform, discuss, decide, or brainstorm.")
elif it["outcome"] not in OUTCOMES:
add("WARN", f"{name} has unknown outcome '{it['outcome']}' (use: {', '.join(sorted(OUTCOMES))}).")
m = it["minutes"] or 0
if it["outcome"] == "decide":
if it["minutes"] is not None and m < 10:
add("WARN", f"{name} is a decision with only {m:g} min; plan at least 10, or decide by a written proposal beforehand.")
if it["start"] - cfg["start"] >= 0.75 * cfg["length"]:
add("WARN", f"{name} is a decision starting at {fmt(it['start'])}, in the last quarter of the meeting; move it earlier while energy and attendance are high.")
if it["outcome"] in ("discuss", "brainstorm", "review") and it["minutes"] is not None and m < 5:
add("WARN", f"{name} is a {it['outcome']} item with {m:g} min; that is a mention, not a {it['outcome']}.")
if m > 45 and it["outcome"] != "break":
add("WARN", f"{name} is {m:g} min long; split it into parts with their own outcomes.")
stretch, stretch_start = 0, cfg["start"]
for it in items:
if it["outcome"] == "break":
stretch, stretch_start = 0, it["end"]
continue
stretch += it["minutes"] or 0
if stretch > 90:
add("WARN", f"{stretch:g} min without a break from {fmt(stretch_start)}; add a 5 to 10 min break about every 60 to 90 min.")
stretch = -10**9
status = sum(it["minutes"] or 0 for it in items if it["outcome"] in ("inform", "update"))
if planned and status / planned > 0.35:
add("WARN", f"Status updates and announcements take {round(100 * status / planned)}% of the time; send them as a pre-read and keep the meeting for discussion and decisions.")
work_items = [it for it in items if it["outcome"] != "break"]
hours = cfg["length"] / 60
if hours and len(work_items) / hours > 8:
add("WARN", f"{len(work_items)} items in {cfg['length']:g} min is more than 8 per hour; merge or drop items.")
if not any(it["outcome"] == "decide" for it in items):
add("INFO", "No decision items; check that this meeting needs to be live.")
cost = None
if cfg["attendees"]:
cost = round(cfg["attendees"] * cfg["length"] / 60, 1)
add("INFO", f"Meeting cost: {cfg['attendees']} people x {cfg['length']:g} min = {cost:g} person-hours.")
order = {"HIGH": 0, "WARN": 1, "INFO": 2}
findings.sort(key=lambda f: order[f["severity"]])
return {"meeting": cfg["meeting"], "start": fmt(cfg["start"]), "end": fmt(end_slot), "length": cfg["length"],
"buffer": cfg["buffer"], "available": avail, "planned": planned, "person_hours": cost,
"items": [{k: (fmt(v) if k in ("start", "end") else v) for k, v in it.items()} for it in items],
"findings": findings}
def main():
ap = argparse.ArgumentParser(description="Timebox a meeting agenda")
ap.add_argument("path", help="agenda text/Markdown/JSON file, or - for stdin")
ap.add_argument("--fit", action="store_true", help="scale item durations to fit the available time")
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
try:
text = sys.stdin.read() if args.path == "-" else open(args.path, encoding="utf-8").read()
if not text.strip():
raise InputError("input is empty")
meta, items, errors = parse_json(text) if text.lstrip().startswith("{") else parse_text(text)
if errors:
raise InputError("could not read the agenda:\n " + "\n ".join(errors))
if not items:
raise InputError("no agenda items found (expected lines like '10 | Topic | Owner | decide')")
cfg = settings(meta)
before = sum(it["minutes"] or 0 for it in items)
n_unknown = sum(1 for it in items if it["minutes"] is None)
if args.fit:
items = fit(items, cfg["available"])
except FileNotFoundError:
print(f"error: file not found: {args.path}", file=sys.stderr)
return 2
except (InputError, json.JSONDecodeError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
rep = analyze(items, cfg)
rep["fitted_from"] = before if args.fit else None
if args.json:
print(json.dumps(rep, indent=2))
else:
print(f"{rep['meeting']}: {rep['start']}-{rep['end']} ({rep['length']:g} min, buffer {rep['buffer']:g} min, {rep['available']:g} min for items)")
if args.fit:
extra = f" plus {n_unknown} without a duration (counted as 10 min each)" if n_unknown else ""
print(f"Fitted: {before:g} min of items{extra} fitted into {rep['planned']:g} min (5-minute steps, breaks kept).")
print(f"Planned: {rep['planned']:g} min\n")
for it in rep["items"]:
mins = "?" if it["minutes"] is None else f"{it['minutes']:g}"
was = f" (was {it['original']:g})" if args.fit and it.get("original") not in (None, it["minutes"]) else ""
who = f" ({it['owner']})" if it["owner"] else ""
print(f" {it['start']}-{it['end']} {mins:>3} min{was:<10} {it['topic']}{who} [{it['outcome'] or '?'}]")
if rep["buffer"]:
print(f" {fmt(hhmm(rep['end']) - rep['buffer'])}-{rep['end']} {rep['buffer']:>3g} min Buffer / wrap-up")
print()
if rep["findings"]:
print("Findings:")
for f in rep["findings"]:
print(f"- [{f['severity']}] {f['message']}")
else:
print("Findings: none")
return 1 if any(f["severity"] == "HIGH" for f in rep["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())【中文说明】用于检查 SRT 或 WebVTT 字幕的时间轴、编号、阅读速度、行长和标签问题，并修订异常片段。粘贴或上传字幕文件，提供每秒字符数、行数与显示时长标准；如需整体平移时间轴，再注明偏移量。 Checks SRT and WebVTT subtitle files for broken or overlapping timings, numbering gaps, empty cues, reading speed above a characters-per-second limit, long or extra lines, cues that flash by or linger, unbalanced tags, and sound labels, then rewrites and retimes the problem cues and can shift the whole file. Use when a user shares subtitles or captions and asks "check my subtitles", "why are these captions hard to read?", "fix the timing", or "make these subtitles follow the guidelines".
---
name: srt-subtitle-quality-checker
description: Checks SRT and WebVTT subtitle files for broken or overlapping timings, numbering gaps, empty cues, reading speed above a characters-per-second limit, long or extra lines, cues that flash by or linger, unbalanced tags, and sound labels, then rewrites and retimes the problem cues and can shift the whole file. Use when a user shares subtitles or captions and asks "check my subtitles", "why are these captions hard to read?", "fix the timing", or "make these subtitles follow the guidelines".
---
# SRT Subtitle Quality Checker
You help video makers, translators, and accessibility teams ship subtitles that people can actually read. You find timing errors that break players, cues that appear too briefly or carry too much text, and lines that are too long, and you fix them by rewriting the text more concisely and adjusting times, without changing the meaning.
## Files in this skill
- `scripts/check_subtitles.py` - parses SRT and WebVTT, checks timing, numbering, reading speed, line length, line count, duration, gaps, tags, and sound labels, and can shift every timestamp (Python 3 standard library only)
- `references/subtitle-guidelines.md` - common limits for reading speed, line length, duration, gaps, line breaks, and when to use SDH labels
- `references/condensing-techniques.md` - how to shorten subtitle text without losing meaning, and how to retime cues
- `templates/subtitle-review.md` - the review report to return
- `examples/example-workshop-video.md` - a worked review of a short tutorial video
## Workflow
### 1. Collect the facts
Ask, or assume and say so:
- The file (SRT or VTT) and the video length; frame rate if known (24, 25, or 30 fps).
- Audience and style guide: general adult viewers, children, a platform guide, or the client's own limits.
- Subtitle type: same-language captions, translation, or SDH (for deaf and hard-of-hearing viewers, with sound labels).
- Whether you may rewrite text or only retime it.
### 2. Run the checker
```bash
python3 scripts/check_subtitles.py movie.srt
python3 scripts/check_subtitles.py movie.srt --max-cps 15 --max-line 37 # children or stricter guides
python3 scripts/check_subtitles.py movie.vtt --json
python3 scripts/check_subtitles.py movie.srt --shift -1200 > movie-synced.srt # whole file 1.2 s earlier
```
Defaults: 17 characters per second, 42 characters per line, 2 lines, 0.833 to 7 seconds on screen, 83 ms minimum gap. Pick limits from `references/subtitle-guidelines.md` to match the audience. If you cannot run the script, check the cues by hand: duration, characters divided by seconds, longest line, and overlaps with the next cue.
### 3. Fix in this order
1. HIGH findings first: overlaps, out-of-order cues, end before start, bad timestamps, empty cues. These break players or hide text.
2. Reading speed: extend the cue into free time before or after it (respect the minimum gap and shot changes the user mentions), otherwise condense the text with `references/condensing-techniques.md`, otherwise split the cue in two at a natural pause.
3. Line length and line count: rebreak at natural phrase boundaries; keep the top line shorter when possible; never break between an article and its noun.
4. Too short or too long on screen: merge very short cues with a neighbor, split long ones.
5. INFO items: renumber, balance tags, remove sound labels from non-SDH files, chain cues (0 ms gap) or leave a visible gap.
Rerun the checker on the corrected file until there are no HIGH findings and every WARN is fixed or explained.
### 4. Deliver
Fill in `templates/subtitle-review.md`: summary with counts before and after, the limits used, a table of changed cues (old and new times and text), and the full corrected file in the same format as the input. Follow the style of `examples/example-workshop-video.md`.
## Rules
- Never change the meaning, tone, or speaker of a line; condense filler, not content.
- Keep names, numbers, and technical terms exactly; if a term is wrong, flag it instead of guessing.
- Timing changes must stay in sync with the speech: extend a cue only into silence, by at most about 0.5 s before speech starts or 1 s after it ends, unless the user allows more.
- Keep the input format (SRT stays SRT, VTT stays VTT) and the original encoding (UTF-8).
- Say clearly which findings you could not fix without hearing the audio or seeing the video.
FILE:references/subtitle-guidelines.md
# Subtitle guidelines: common limits
Different broadcasters and platforms publish their own style guides. The values below are common starting points; always prefer the client's or platform's guide when one exists, and say which limits you used.
## Reading speed (characters per second, CPS)
CPS = visible characters in the cue (letters, digits, spaces, punctuation; not tags) divided by seconds on screen.
| Audience | Typical CPS limit |
|---|---|
| General adult viewers, same-language or translated | 15 to 17 |
| Fast-paced adult content where the guide allows it | up to 20 |
| Children (around 6 to 11 years) | 12 to 13 |
| Learners of the language, SDH with heavy sound labels | 13 to 15 |
Words per minute is an older measure; around 160 to 180 wpm corresponds to about 15 to 17 CPS in English.
## Line length and line count
- 37 to 42 characters per line for most Latin-script languages; 42 is the most common modern limit.
- Maximum 2 lines per cue. A third line covers the picture and is hard to read.
- Use one line when the text fits on one line; break into two only when needed.
## Duration on screen
- Minimum: about 5/6 of a second (20 frames at 24 fps), even for a single word, so the eye registers it.
- Maximum: about 7 seconds. Longer cues get reread and feel stuck; split them.
## Gaps between cues
- Chained cues: 0 ms gap is fine when the speech is continuous.
- Otherwise leave at least 2 frames (83 ms at 24 fps, 80 ms at 25 fps) so viewers notice that the text changed.
- Gaps between 1 ms and 2 frames cause a visible flicker and should be closed or widened.
## Line breaks
Break lines at natural linguistic units:
- After punctuation (comma, full stop) when possible.
- Before conjunctions (and, but, because) and prepositions.
- Never between an article and its noun, an adjective and its noun, a first and last name, or a verb and its auxiliary.
- Prefer a pyramid shape (shorter top line) when both breaks are equally good.
## Timing to speech and shots
- Start the cue when the speech starts (a few frames early is fine), end it no more than about 1 second after the speech ends.
- Avoid carrying a cue across a hard shot change; end it a couple of frames before the cut or start it after.
## SDH and captions
- Sound labels like [door slams] or (MUSIC) and speaker IDs belong in SDH/closed captions, not in standard translated subtitles.
- Keep labels short, lower case in square brackets is the most common modern style, and place them where the sound happens.
## Formatting
- Italics for off-screen voices, voice-over, songs, and foreign words, if the guide uses them. Every opening tag needs a closing tag in the same cue.
- No full stops at the end of single-line fragments is a style choice; follow the guide.
- SRT: cue number, timestamp line `00:01:02,500 --> 00:01:04,000`, text, blank line. VTT: `WEBVTT` header, optional cue id, timestamps with a dot `00:01:02.500`.
FILE:references/condensing-techniques.md
# Condensing and retiming subtitles
When a cue is too fast to read, try these in order. Stop as soon as the cue fits the limits.
## 1. Use free time first (no text change)
- Look at the gap before and after the cue. Extend the start up to about 0.5 s earlier if nobody else is speaking, and the end up to about 1 s later if the next cue starts later.
- Keep the minimum gap (2 frames) to the neighbors, or chain at 0 ms.
- Required time for a cue = characters / CPS limit. Example: 68 characters at 17 CPS need 4.0 s.
## 2. Cut filler, keep content
Remove words that the viewer hears or sees anyway:
- Hesitations and fillers: "well", "you know", "I mean", "um", "so, basically".
- Repetitions: "very, very" -> "very"; false starts.
- Greetings and names already obvious from the picture.
- Tag questions when tone is clear: "It's ready, isn't it?" -> "It's ready?" only if the meaning stays.
## 3. Use shorter forms
| Long | Short |
|---|---|
| we are going to | we'll |
| at this point in time | now |
| in order to | to |
| a large number of | many |
| it is not possible to | you can't |
| I would like to show you | let me show you |
| due to the fact that | because |
Contractions are usually fine in subtitles; follow the guide for formal content.
## 4. Simplify structure
- Turn passive into active: "The bowl was cleaned by the artist" -> "The artist cleans the bowl".
- Replace a clause with a single word when possible.
- Split one long sentence into two shorter cues at a natural pause.
## 5. Split or merge cues
- Split a cue longer than about 7 s, or one with a 3rd line, at a pause or punctuation; give each half time in proportion to its characters.
- Merge a cue under about 0.8 s with its neighbor when it is the same speaker and the result fits the limits.
## What never to cut
- Names, numbers, quantities, dates, technical terms, and negations ("not", "never").
- Words that carry the joke, the twist, or the emotion of the line.
- Information the viewer needs later in the video.
## Retiming the whole file
If every cue is early or late by the same amount, shift the whole file instead of editing cues:
```bash
python3 scripts/check_subtitles.py movie.srt --shift 1500 > movie-shifted.srt # 1.5 s later
python3 scripts/check_subtitles.py movie.srt --shift -800 > movie-shifted.srt # 0.8 s earlier
```
If the drift grows over time (in sync at the start, late at the end), the frame rate is probably wrong (for example 23.976 vs 25 fps); a constant shift will not fix it. Say so and ask for the frame rate of the video.
FILE:templates/subtitle-review.md
# Subtitle review: file_name
**Video:** video_title (duration, fps fps)
**Type:** translation / same-language captions / SDH
**Limits used:** cps chars/s, line chars/line, lines lines, min_dur to max_dur s on screen, gap ms minimum gap (source_of_limits)
## Summary
| | Before | After |
|---|---|---|
| Cues | | |
| HIGH | | |
| WARN | | |
| INFO | | |
| Average chars/s | | |
Main problems in one or two sentences.
## Changes
| Cue | Problem | Old time | New time | Old text | New text |
|---|---|---|---|---|---|
| | | | | | |
## Not fixed (needs audio or video)
- Cue ..., reason ...
## Notes for the editor
- Style choices made (italics, contractions, labels).
- Anything to check against the picture (shot changes, on-screen text).
## Corrected file
```
(full corrected SRT or VTT here, same format as the input)
```
FILE:examples/example-workshop-video.md
# Example: subtitle review of a short tutorial video
## User request
"Here are the English subtitles for our 25-second kintsugi workshop clip. Viewers say they can't keep up. Can you check them and fix what's wrong? General adult audience, normal subtitles, not SDH."
## Input (excerpt of interview.srt)
```
2
00:00:03,250 --> 00:00:04,000
Today we are going to look at how to repair a cracked ceramic bowl with gold.
3
00:00:03,900 --> 00:00:06,500
[HAMMER TAPPING]
First, clean the edges.
5
00:00:07,000 --> 00:00:16,000
<i>This technique is called kintsugi.
```
## Checker run
```
$ python3 scripts/check_subtitles.py interview.srt
interview.srt (SRT): 8 cues, 00:00:01,000 to 00:00:24,000, avg 16.3 chars/s, avg 2.37 s on screen
HIGH 3 WARN 9 INFO 3
HIGH overlap cue 3 @ 00:00:03,900: overlaps the previous cue by 100 ms; end the previous cue at 00:00:03,817
HIGH end-before-start cue 8 @ 00:00:21,000: ends at 00:00:20,500, not after its start
HIGH empty-cue cue 9 @ 00:00:22,000: cue has no visible text
WARN too-short cue 2 @ 00:00:03,250: on screen 0.75 s (minimum 0.83 s)
WARN reading-speed cue 2 @ 00:00:03,250: 102.7 chars/s (limit 17); needs 4.53 s on screen or about 64 fewer chars
...
```
## Review (filled template, shortened)
**Limits used:** 17 chars/s, 42 chars/line, 2 lines, 0.833 to 7 s, 83 ms gap (common general-audience values; no client guide given).
| | Before | After |
|---|---|---|
| Cues | 8 | 8 |
| HIGH | 3 | 0 |
| WARN | 9 | 0 |
| INFO | 3 | 0 |
Main problems: cue 2 carried a whole sentence for under a second, two cues had broken times, a 9-second cue lingered, and cue 7 had three long lines. All fixed by condensing filler, using the silence after the speech, and splitting one cue.
| Cue | Problem | Old time | New time | Old text | New text |
|---|---|---|---|---|---|
| 2 | 102.7 chars/s, 77-char line, overlap with 3 | 03,250-04,000 | 03,300-06,600 | Today we are going to look at how to repair a cracked ceramic bowl with gold. | Today we'll repair a cracked bowl / with lacquer and gold. |
| 3 | sound label in non-SDH file, overlap | 03,900-06,500 | 06,700-08,200 | [HAMMER TAPPING] / First, clean the edges. | First, clean the edges. |
| 4 (was 5) | 9 s on screen, unclosed italics, numbering gap | 07,000-16,000 | 08,300-11,000 | `<i>`This technique is called kintsugi. | `<i>`This technique is called kintsugi.`</i>` |
| 5 (was 6) | 0.4 s on screen | 16,000-16,400 | 16,000-16,900 | Okay. | Okay. |
| 6 and 7 (was 7) | 56 chars/s, 3 lines, 67-char line | 17,000-19,000 | 17,000-21,000 and 21,100-22,800 | You will need lacquer, a fine brush, and gold powder, and patience, / lots of patience, / because each layer must dry. | You'll need lacquer, a fine brush, / gold powder, and lots of patience. + Each layer must dry first. |
| 8 (was 8) | ends before it starts | 21,000-20,500 | 22,900-24,000 | Let's start. | Let's start. |
| 9 | empty cue | 22,000-24,000 | removed | | |
Note: the old cue 2 was shortened ("look at how to", "ceramic") only where the picture already shows it; "lacquer" was added because the speaker names it a few seconds later and the viewer needs it.
**Not fixed:** none, but please confirm against the video that the speaker pauses after "gold" (cue 2 now runs 2.6 s longer, into what looks like silence) and that "Let's start." is spoken at about 22.9 s.
Corrected file rechecked:
```
$ python3 scripts/check_subtitles.py interview-fixed.srt
interview-fixed.srt (SRT): 8 cues, 00:00:01,000 to 00:00:24,000, avg 14.5 chars/s, avg 2.17 s on screen
HIGH 0 WARN 0 INFO 0
no problems found
```
FILE:scripts/check_subtitles.py
#!/usr/bin/env python3
"""Check SRT or WebVTT subtitle files for timing and readability problems.
Usage:
python3 check_subtitles.py FILE [FILE ...] [options]
python3 check_subtitles.py - < movie.srt (read stdin)
Options:
--max-cps N maximum characters per second (default 17)
--max-line N maximum characters per line (default 42)
--max-lines N maximum lines per cue (default 2)
--min-dur S minimum cue duration in seconds (default 0.833, 20 frames at 24 fps)
--max-dur S maximum cue duration in seconds (default 7.0)
--min-gap MS minimum gap between cues in milliseconds (default 83, 2 frames at 24 fps)
--shift MS print the file with every timestamp shifted by MS (may be negative) and exit
--json print findings as JSON
--fail-on LEVEL exit 1 if any finding is at least LEVEL: high (default), warn, info
Findings: HIGH = broken or overlapping timing, unreadable numbering, empty cue;
WARN = reading speed, line length, too many lines, too short or long on screen;
INFO = tiny gaps, unbalanced tags, hearing-impaired brackets, statistics.
Exit codes: 0 ok, 1 findings at or above --fail-on, 2 usage or parse error.
Python 3 standard library only.
"""
import json
import math
import re
import sys
TS = re.compile(r"^\s*(\d{1,2}:)?(\d{1,2}):(\d{2})[,.](\d{1,3})\s*-->\s*(\d{1,2}:)?(\d{1,2}):(\d{2})[,.](\d{1,3})(.*)$")
TAG = re.compile(r"</?[^>]+>|\{\\[^}]*\}")
LEVELS = {"info": 0, "warn": 1, "high": 2}
def to_ms(h, m, s, ms):
h = int(h[:-1]) if h else 0
return ((h * 60 + int(m)) * 60 + int(s)) * 1000 + int(ms.ljust(3, "0"))
def fmt(ms, vtt=False):
sign = "-" if ms < 0 else ""
ms = abs(ms)
h, rem = divmod(ms, 3600000)
m, rem = divmod(rem, 60000)
s, rem = divmod(rem, 1000)
sep = "." if vtt else ","
return f"{sign}{h:02d}:{m:02d}:{s:02d}{sep}{rem:03d}"
def parse(text):
"""Return (cues, errors, is_vtt). Each cue: dict(index, start, end, lines, line_no)."""
text = text.replace("\r\n", "\n").replace("\r", "\n").lstrip("\ufeff")
is_vtt = text.lstrip().startswith("WEBVTT")
blocks = re.split(r"\n\s*\n", text.strip("\n"))
cues, errors = [], []
line_no = 1
for block in blocks:
lines = block.split("\n")
start_line = line_no
line_no += len(lines) + 1
if is_vtt and (lines[0].startswith("WEBVTT") or lines[0].startswith("NOTE") or lines[0].startswith("STYLE") or lines[0].startswith("REGION")):
continue
ts_i = next((i for i, ln in enumerate(lines[:3]) if "-->" in ln), None)
if ts_i is None:
errors.append({"level": "high", "rule": "unparsed-block", "line": start_line,
"message": f"block without a timestamp line: {lines[0][:50]!r}"})
continue
m = TS.match(lines[ts_i])
if not m:
errors.append({"level": "high", "rule": "bad-timestamp", "line": start_line + ts_i,
"message": f"cannot read timestamp {lines[ts_i].strip()!r} (expected 00:01:02,500 --> 00:01:04,000)"})
continue
g = m.groups()
index = lines[0].strip() if ts_i == 1 else None
cues.append({"index": index, "start": to_ms(*g[0:4]), "end": to_ms(*g[4:8]),
"lines": [ln for ln in lines[ts_i + 1:]], "line_no": start_line})
return cues, errors, is_vtt
def visible(line):
return TAG.sub("", line).strip()
def check(cues, errors, opts, is_vtt):
out = list(errors)
def add(level, rule, cue, msg):
out.append({"level": level, "rule": rule, "line": cue["line_no"],
"cue": cue["index"] or "-", "time": fmt(cue["start"], is_vtt), "message": msg})
if not is_vtt:
expected = 1
for c in cues:
if c["index"] is None or not c["index"].isdigit():
add("high", "bad-number", c, f"cue number missing or not a number: {c['index']!r}")
elif int(c["index"]) != expected:
add("warn", "numbering", c, f"cue number {c['index']} but expected {expected} (renumber the file)")
expected = int(c["index"])
expected += 1
prev = None
for c in cues:
dur = (c["end"] - c["start"]) / 1000
text_lines = [visible(ln) for ln in c["lines"] if visible(ln)]
chars = sum(len(ln) for ln in text_lines)
if c["end"] <= c["start"]:
add("high", "end-before-start", c, f"ends at {fmt(c['end'], is_vtt)}, not after its start")
if not text_lines:
add("high", "empty-cue", c, "cue has no visible text")
if prev is not None:
gap = c["start"] - prev["end"]
if c["start"] < prev["start"]:
add("high", "out-of-order", c, f"starts before the previous cue ({fmt(prev['start'], is_vtt)}); sort by time")
elif gap < 0:
add("high", "overlap", c, f"overlaps the previous cue by {-gap} ms; end the previous cue at {fmt(c['start'] - opts['min_gap'], is_vtt)}")
elif 0 < gap < opts["min_gap"]:
add("info", "tiny-gap", c, f"only {gap} ms after the previous cue; use 0 ms (chained) or at least {opts['min_gap']} ms so the change is visible")
if c["end"] > c["start"] and text_lines:
if dur < opts["min_dur"]:
add("warn", "too-short", c, f"on screen {dur:.2f} s (minimum {opts['min_dur']:.2f} s)")
if dur > opts["max_dur"]:
add("warn", "too-long", c, f"on screen {dur:.2f} s (maximum {opts['max_dur']:.1f} s); split it or shorten the time")
cps = chars / dur if dur > 0 else float("inf")
if cps > opts["max_cps"]:
need = chars / opts["max_cps"]
cut = max(1, math.ceil(chars - opts["max_cps"] * dur))
add("warn", "reading-speed", c, f"{cps:.1f} chars/s (limit {opts['max_cps']:g}); needs {need:.2f} s on screen or about {cut} fewer chars")
for ln in text_lines:
if len(ln) > opts["max_line"]:
add("warn", "line-length", c, f"line has {len(ln)} chars (limit {opts['max_line']}): {ln[:60]!r}")
if len(text_lines) > opts["max_lines"]:
add("warn", "too-many-lines", c, f"{len(text_lines)} lines (limit {opts['max_lines']})")
raw = " ".join(c["lines"])
for tag in ("i", "b", "u"):
if len(re.findall(rf"<{tag}>", raw, re.I)) != len(re.findall(rf"</{tag}>", raw, re.I)):
add("info", "unbalanced-tag", c, f"<{tag}> tags are not balanced in this cue")
if re.search(r"\[[^\]]+\]|\([A-Z][A-Z ]+\)", raw):
add("info", "sdh-label", c, "contains a sound or speaker label in brackets; keep it only in SDH/CC files")
prev = c
return out
def stats(cues, is_vtt=False):
if not cues:
return {}
total_chars = sum(len(visible(ln)) for c in cues for ln in c["lines"])
total_time = sum(max(c["end"] - c["start"], 0) for c in cues) / 1000
return {"cues": len(cues), "first": fmt(cues[0]["start"], is_vtt), "last_end": fmt(max(c["end"] for c in cues), is_vtt),
"avg_cps": round(total_chars / total_time, 1) if total_time else None,
"avg_duration_s": round(total_time / len(cues), 2)}
def shift_text(text, delta):
def rep(m):
g = m.groups()
vtt = "." in m.group(0).split("-->")[0]
a = max(to_ms(*g[0:4]) + delta, 0)
b = max(to_ms(*g[4:8]) + delta, 0)
return f"{fmt(a, vtt)} --> {fmt(b, vtt)}{g[8]}"
return "\n".join(TS.sub(rep, ln) if "-->" in ln else ln
for ln in text.replace("\r\n", "\n").split("\n"))
def main(argv):
opts = {"max_cps": 17.0, "max_line": 42, "max_lines": 2, "min_dur": 0.833, "max_dur": 7.0,
"min_gap": 83, "json": False, "fail_on": "high", "shift": None}
files = []
it = iter(argv)
try:
for a in it:
if a == "--json":
opts["json"] = True
elif a in ("--max-cps", "--min-dur", "--max-dur"):
opts[a[2:].replace("-", "_")] = float(next(it))
elif a in ("--max-line", "--max-lines", "--min-gap", "--shift"):
opts[a[2:].replace("-", "_")] = int(next(it))
elif a == "--fail-on":
opts["fail_on"] = next(it).lower()
if opts["fail_on"] not in LEVELS:
raise ValueError("--fail-on must be high, warn, or info")
elif a in ("-h", "--help"):
print(__doc__)
return 0
elif a.startswith("--"):
raise ValueError(f"unknown option {a}")
else:
files.append(a)
except (StopIteration, ValueError) as e:
print(f"error: {e or 'option needs a value'}", file=sys.stderr)
return 2
if not files:
print("error: give at least one .srt or .vtt file, or - for stdin", file=sys.stderr)
return 2
report, worst = [], -1
for f in files:
try:
text = sys.stdin.read() if f == "-" else open(f, encoding="utf-8-sig").read()
except (OSError, UnicodeDecodeError) as e:
print(f"error: cannot read {f}: {e}", file=sys.stderr)
return 2
if opts["shift"] is not None:
sys.stdout.write(shift_text(text, opts["shift"]))
return 0
cues, errors, is_vtt = parse(text)
if not cues:
print(f"error: {f}: no subtitle cues found (is this an SRT or WebVTT file?)", file=sys.stderr)
return 2
findings = check(cues, errors, opts, is_vtt)
for x in findings:
worst = max(worst, LEVELS[x["level"]])
report.append({"file": f, "format": "vtt" if is_vtt else "srt", "stats": stats(cues, is_vtt), "findings": findings})
if opts["json"]:
print(json.dumps(report, indent=2))
else:
for r in report:
counts = {lv: sum(1 for x in r["findings"] if x["level"] == lv) for lv in ("high", "warn", "info")}
st = r["stats"]
print(f"{r['file']} ({r['format'].upper()}): {st['cues']} cues, {st['first']} to {st['last_end']}, "
f"avg {st['avg_cps']} chars/s, avg {st['avg_duration_s']} s on screen")
print(f" HIGH {counts['high']} WARN {counts['warn']} INFO {counts['info']}")
for x in sorted(r["findings"], key=lambda x: (-LEVELS[x["level"]], x["line"])):
where = f"cue {x.get('cue', '-')} @ {x['time']}" if "time" in x else f"line {x['line']}"
print(f" {x['level'].upper():<4} {x['rule']:<16} {where}: {x['message']}")
if not r["findings"]:
print(" no problems found")
return 1 if worst >= LEVELS[opts["fail_on"]] else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))【中文说明】用于计算固定利率贷款的还款明细，并比较额外月供、一次性提前还款或缩短期限的效果。提供本金、利率、期限、起始日期与费用，再说明拟采用的还款策略，可查看利息节省、还清时间和费用相关估算。 Builds fixed-rate loan and mortgage amortization schedules, shows how much of each payment goes to interest, and compares payoff strategies such as extra monthly payments, one-off lump sums, a higher fixed payment, or a shorter term, with interest saved, months saved, payoff date, and an APR estimate when fees apply. Use when a user asks "how much interest will I pay?", "should I overpay my mortgage?", "what if I pay 200 more a month?", or wants a payoff plan for a car, student, or home loan.
--- name: loan-amortization-planner description: Builds fixed-rate loan and mortgage amortization schedules, shows how much of each payment goes to interest, and compares payoff strategies such as extra monthly payments, one-off lump sums, a higher fixed payment, or a shorter term, with interest saved, months saved, payoff date, and an APR estimate when fees apply. Use when a user asks "how much interest will I pay?", "should I overpay my mortgage?", "what if I pay 200 more a month?", or wants a payoff plan for a car, student, personal, or home loan. Includes a tested stdlib Python calculator. --- # Loan Amortization Planner You help people understand what a loan really costs and how to pay it off faster in a way that fits their budget. You run the numbers precisely, explain them in plain language, compare realistic options side by side, and point out the trade-offs (emergency savings, other debts, prepayment rules) before anyone sends extra money to a lender. ## Files in this skill - `scripts/amortize.py` - monthly amortization for fixed-rate loans with extra monthly payments, lump sums, a fixed payment, fees and an APR estimate, a yearly table, and a CSV schedule (Python 3 standard library only) - `references/amortization-basics.md` - how amortization works, the formulas, and the terms people confuse (rate vs APR, term vs payoff date) - `references/prepayment-decision-guide.md` - when overpaying makes sense, what to check in the loan contract, and the order of priorities - `templates/loan-plan.md` - the report to return - `examples/example-mortgage-overpayment.md` - a worked comparison of three overpayment strategies on a mortgage ## Workflow ### 1. Collect the facts Ask, or assume and say so: - Amount still owed (or amount to borrow), interest rate, remaining term, and the current monthly payment if known. - Fixed or variable rate, and when a fixed period ends. - Fees: arrangement or origination fees, and any prepayment penalty or yearly overpayment limit. - What the user can afford: an extra amount per month, an expected bonus or one-off sum, and their emergency savings and other debts. - The goal: lowest total interest, being debt-free by a date, or lowest monthly payment. ### 2. Run the baseline ```bash python3 scripts/amortize.py --principal 300000 --rate 4.5 --years 30 --start 2026-12 python3 scripts/amortize.py --principal 18500 --rate 7.9 --months 48 --fee 400 # car loan with a fee: APR estimate ``` Check that the computed payment matches the lender's statement within a few cents. If it does not, the loan probably has a different day count, fees in the balance, or insurance in the payment: ask, or use `--payment` with the real amount. ### 3. Run the scenarios ```bash python3 scripts/amortize.py --principal 300000 --rate 4.5 --years 30 --extra 200 python3 scripts/amortize.py --principal 300000 --rate 4.5 --years 30 --lump 12:10000 --lump 24:10000 python3 scripts/amortize.py --principal 300000 --rate 4.5 --years 30 --payment 2000 --yearly python3 scripts/amortize.py --principal 300000 --rate 4.5 --years 20 # shorter term python3 scripts/amortize.py ... --schedule plan.csv --json ``` Compare 2 to 4 options that the user can really afford. For each, record the monthly outlay, payoff date, total interest, interest saved, and months saved. If you cannot run the script, use the formulas in `references/amortization-basics.md` and say the numbers are approximate. ### 4. Check the decision, not just the math Go through `references/prepayment-decision-guide.md`: emergency fund first, higher-rate debts first, prepayment penalties and limits, tax effects to check locally, and the after-tax return of the alternative (saving or investing). Never present overpaying as always right. ### 5. Deliver Fill in `templates/loan-plan.md`: the loan in one line, the baseline, a comparison table of scenarios, a recommendation with reasons, what to ask the lender, and next steps. Follow the style of `examples/example-mortgage-overpayment.md`. ## Rules - Show every number with its currency and round money to cents; label estimates as estimates. - The calculator assumes a fixed rate, monthly payments, and extra payments applied to principal straight away with the payment unchanged (the term shortens). Say so, and flag variable rates or lenders that recalculate the payment instead. - Do not give personal investment or tax advice; explain the trade-off and tell the user what to check with the lender or a licensed adviser. - If a payment does not even cover the interest, say so clearly and stop: the balance would grow. - Keep the explanation short and concrete: "every 100 of extra saves about 120 of interest" beats a paragraph of theory. FILE:references/amortization-basics.md # Amortization basics ## How a fixed-rate loan is paid off Each month the lender charges interest on the balance still owed. Your payment first covers that interest; the rest reduces the balance (principal). Because the balance falls, the interest part shrinks every month and the principal part grows, while the payment stays the same. Early payments are mostly interest; late payments are mostly principal. ## Formulas - Monthly rate: r = annual rate / 12 (for example 4.5% -> 0.00375). - Payment for a loan P over n months: M = P * r / (1 - (1 + r)^-n). With r = 0: M = P / n. - Interest this month: balance * r. Principal this month: M - interest. - Total interest: sum of the interest parts, or approximately M * n - P. - Months to pay off with payment M: n = -ln(1 - P * r / M) / ln(1 + r). If P * r >= M the loan never ends. Quick checks: 200,000 at 6% for 30 years -> 1,199.10 per month. 300,000 at 4.5% for 30 years -> 1,520.06 per month. ## Why extra payments save so much An extra payment reduces the balance immediately, so every later month charges interest on a smaller amount. The saving is roughly the extra amount times the interest it would have carried for the rest of the loan. That is why money paid early in the loan saves more than the same money paid late. ## Terms people confuse | Term | Meaning | |---|---| | Nominal rate | The yearly rate in the contract, used for the monthly interest. | | APR | Annual percentage rate including fees; compares offers. A loan with a lower rate but high fees can have a higher APR. | | Term | The planned length (e.g. 30 years). | | Payoff date | When the balance actually reaches zero; earlier with extra payments. | | Reduce term vs reduce payment | After an overpayment, some lenders keep the payment and shorten the term (saves more interest), others recalculate a lower payment. Ask which. | | Principal | The amount still owed, not counting future interest. | | Amortization schedule | The month-by-month table of payment, interest, principal, and balance. | ## Rounding and small differences Lenders round to cents each month and may use daily interest (actual days / 365) instead of rate / 12. Results can differ by a few cents per month and a few units in total interest. The last payment usually absorbs the rounding difference. ## Limits of a fixed-rate calculator - Variable or tracker rates: run scenarios at the current rate and at +1 and +2 percentage points. - Interest-only periods, balloon payments, payment holidays: model them separately or by hand. - Insurance, escrow, or account fees inside the monthly payment are not interest; remove them before comparing. FILE:references/prepayment-decision-guide.md # Should I pay off my loan faster? A decision guide Overpaying a loan gives a guaranteed, risk-free "return" equal to the loan's interest rate. That is good, but it is not always the best use of spare money. Work through these steps in order. ## 1. Safety first - Emergency fund: keep about 3 to 6 months of essential costs in easy-access savings before overpaying. Money sent to a lender is hard to get back. - Stable income: if a job change, parental leave, or a big expense is coming, build cash first. ## 2. Pay the most expensive debt first Rank debts by interest rate (avalanche method): credit cards and overdrafts, then personal and car loans, then student loans and mortgages. Overpay the highest rate first. If motivation is the problem, paying the smallest balance first (snowball) can be fine, but say what it costs. ## 3. Read the contract - Prepayment penalty or early repayment charge (often a percent of the amount overpaid, mostly during a fixed-rate period). - Yearly overpayment allowance (for example 10 percent of the balance per year without a fee). - Does an overpayment reduce the term or the payment? Can you choose? - Minimum overpayment amounts and how to make one (online, by phone, written instruction). ## 4. Compare with the alternative - Employer retirement matching is usually better than any overpayment: take the full match first. - Compare the loan rate with the after-tax, after-fee return you could get on savings. A 2% mortgage versus a 4% savings account: saving wins on paper. A 7% car loan versus a 3% savings account: overpaying wins. - Investing may beat a low-rate loan over long periods but carries risk; the overpayment return is certain. - Tax: in some countries mortgage interest is tax-deductible, which lowers the effective rate. Check locally; do not assume. ## 5. Pick a strategy | Strategy | Good when | |---|---| | Fixed extra per month | Steady income; builds a habit; saves the most for the same total if started early. | | Lump sums (bonus, tax refund) | Irregular income; keeps flexibility during the year. | | Higher fixed payment | Wants a clear debt-free date. | | Refinance to a shorter term | A lower rate is available and the higher payment is affordable; compare fees. | | Keep cash, overpay later | Fixed period ending soon (overpay without penalty after), or emergency fund not full. | ## 6. Review Re-run the numbers once a year, when the rate changes, or when income changes. Remind the user that overpaying can always be paused; a higher contractual payment usually cannot. FILE:templates/loan-plan.md # Loan plan: loan_name **Loan:** balance at rate% (fixed_or_variable), remaining_term left, payment payment per month, first payment in this plan start_month **Fees and rules:** fees; prepayment limit or penalty: prepayment_rules **Goal:** goal **Budget for extra payments:** extra_budget ## Baseline - Monthly payment: - Total interest from now: - Payoff date: - First payment split: interest ... / principal ... ## Scenarios | Option | Monthly outlay | Payoff date | Total interest | Interest saved | Months saved | |---|---|---|---|---|---| | Baseline | | | | - | - | | A: | | | | | | | B: | | | | | | | C: | | | | | | ## Recommendation Option ... because: 1. 2. 3. Trade-offs to keep in mind: ## Before you overpay - [ ] Emergency fund covers ... months - [ ] No higher-rate debt left (or it comes first) - [ ] Prepayment penalty / allowance checked with the lender - [ ] Overpayment set to reduce the term (or payment, if preferred) ## Questions for the lender 1. 2. ## Next steps 1. 2. *Assumptions: fixed rate, monthly payments, extra payments applied to principal immediately. These are estimates, not financial advice.* FILE:examples/example-mortgage-overpayment.md # Example: comparing three mortgage overpayment strategies ## User request "We have 320,000 left on our mortgage at 4.1% fixed with 25 years to go, first payment of this plan in December 2026. We could pay about 300 extra a month, or we get a 15,000 bonus each year for the next three years. A friend says we should just switch to 2,400 a month. What's best? We have 6 months of expenses saved and no other debt. The bank allows 10% overpayment per year without fees." ## Commands ```bash python3 scripts/amortize.py --principal 320000 --rate 4.1 --years 25 --start 2026-12 python3 scripts/amortize.py --principal 320000 --rate 4.1 --years 25 --start 2026-12 --extra 300 python3 scripts/amortize.py --principal 320000 --rate 4.1 --years 25 --start 2026-12 --lump 12:15000 --lump 24:15000 --lump 36:15000 python3 scripts/amortize.py --principal 320000 --rate 4.1 --years 25 --start 2026-12 --payment 2400 ``` Output (shortened): ``` Baseline: 1,706.80 per month, interest 192,038.38, total paid 512,038.38, paid off 2051-11 (300 payments) Plan (+300.00/month): paid off 2046-02 (231 payments), interest 143,068.39 Saves 48,969.99 interest and 69 months (5 y 9 m); every 100 of extra saves 70.97 Plan (lump sums 15,000.00 in payment 12, 15,000.00 in payment 24, 15,000.00 in payment 36): paid off 2046-11 (240 payments), interest 133,018.73 Saves 59,019.65 interest and 60 months (5 y 0 m); every 100 of extra saves 131.15 Plan (fixed payment 2,400.00): paid off 2041-10 (179 payments), interest 107,805.37 Saves 84,233.01 interest and 121 months (10 y 1 m) ``` ## Loan plan (filled template, shortened) **Loan:** 320,000.00 at 4.1% fixed, 25 years left, payment 1,706.80 per month, first payment December 2026 **Fees and rules:** overpayments up to 10% of the balance per year are free **Goal:** pay less interest without losing flexibility | Option | Monthly outlay | Payoff date | Total interest | Interest saved | Months saved | |---|---|---|---|---|---| | Baseline | 1,706.80 | 2051-11 | 192,038.38 | - | - | | A: +300 per month | 2,006.80 | 2046-02 | 143,068.39 | 48,969.99 | 69 | | B: 15,000 bonus in years 1 to 3 | 1,706.80 + 45,000 total | 2046-11 | 133,018.73 | 59,019.65 | 60 | | C: fixed 2,400 per month | 2,400.00 | 2041-10 | 107,805.37 | 84,233.01 | 121 | **Recommendation:** Option B, plus A if the budget allows. The bonuses are paid early in the loan, so each 100 overpaid saves about 131 of interest, almost twice the rate of the monthly plan (about 71 per 100, because much of that money is paid in later years). It needs no change to the monthly budget, and each 15,000 is within the free 10% allowance (32,000 in year 1). Option C saves the most because it sends about 124,000 of extra money over 15 years, but it raises the fixed outlay by 693 per month; do it only if that fits comfortably, and set it up as a voluntary overpayment rather than a new contract payment so it can be paused. **Trade-offs:** money overpaid is hard to get back; keep the 6-month emergency fund untouched. If your savings account pays more than 4.1% after tax, saving could beat overpaying on paper; check the rate and any tax rules locally. **Questions for the lender:** 1. Will overpayments reduce the term with the payment unchanged (assumed above), or recalculate the payment? 2. Does the 10% allowance reset per calendar year or per loan year? *Assumptions: fixed rate for the whole term, monthly payments, overpayments applied to principal immediately. Estimates, not financial advice.* FILE:scripts/amortize.py #!/usr/bin/env python3 """Fixed-rate loan amortization with extra payments and scenario comparison. Usage: python3 amortize.py --principal 250000 --rate 4.2 --years 30 [options] python3 amortize.py --principal 18000 --rate 7.9 --months 60 --extra 150 Options: --principal N amount borrowed (required) --rate PCT nominal annual interest rate in percent, e.g. 4.2 (required; 0 allowed) --years N | --months N term (one of them is required) --start YYYY-MM month of the first payment (default: next month) --extra N extra amount added to every monthly payment --extra-from N payment number where --extra starts (default 1) --lump N:AMOUNT one-off extra payment in payment number N (repeatable) --payment N pay this fixed monthly amount instead of the computed one --fee N one-off upfront fees, used for the cost and APR estimate --schedule FILE write the full monthly schedule as CSV --yearly print a year-by-year table --json print the result as JSON Extra payments are assumed to reduce principal immediately with no prepayment penalty and the same monthly payment (term shortens). Rates are nominal annual, compounded monthly (rate / 12 per month). Money is rounded to cents each month. Exit codes: 0 ok, 2 usage or input error. Python 3 standard library only. """ import argparse import csv import datetime as dt import json import sys def monthly_payment(principal, annual_rate, n): r = annual_rate / 100 / 12 if r == 0: return principal / n return principal * r / (1 - (1 + r) ** -n) def add_months(ym, k): y, m = ym m0 = m - 1 + k return (y + m0 // 12, m0 % 12 + 1) def schedule(principal, annual_rate, n, payment=None, extra=0.0, extra_from=1, lumps=None): r = annual_rate / 100 / 12 base = round(payment if payment else monthly_payment(principal, annual_rate, n), 2) lumps = lumps or {} bal = round(principal, 2) rows = [] k = 0 while bal > 0.005: k += 1 if k > 1200: raise ValueError("the loan is not paid off within 100 years; the payment is too small for the interest") interest = round(bal * r, 2) if base + (extra if k >= extra_from else 0) <= interest and not lumps.get(k): raise ValueError(f"payment {base:.2f} does not cover the monthly interest {interest:.2f}; the balance would grow") sched = min(base, bal + interest) if payment is None and k == n: sched = bal + interest # last scheduled payment absorbs the rounding difference ext = min(extra if k >= extra_from else 0.0, max(bal + interest - sched, 0)) lump = min(lumps.get(k, 0.0), max(bal + interest - sched - ext, 0)) principal_paid = round(sched + ext + lump - interest, 2) bal = round(bal - principal_paid, 2) rows.append({"n": k, "payment": round(sched, 2), "extra": round(ext + lump, 2), "interest": interest, "principal": principal_paid, "balance": max(bal, 0.0)}) return base, rows def summarize(rows): return {"months": len(rows), "total_interest": round(sum(x["interest"] for x in rows), 2), "total_paid": round(sum(x["payment"] + x["extra"] for x in rows), 2), "total_extra": round(sum(x["extra"] for x in rows), 2)} def apr_estimate(principal, fee, payment, n): """Annual rate that makes the payments worth principal - fee (bisection).""" target = principal - fee lo, hi = 0.0, 1.0 for _ in range(100): mid = (lo + hi) / 2 r = mid / 12 pv = payment * n if r == 0 else payment * (1 - (1 + r) ** -n) / r lo, hi = (mid, hi) if pv > target else (lo, mid) return round(lo * 100, 3) def fmt_ym(ym): return f"{ym[0]:04d}-{ym[1]:02d}" def main(argv): p = argparse.ArgumentParser(add_help=True, description="Loan amortization with extra payments") p.add_argument("--principal", type=float, required=True) p.add_argument("--rate", type=float, required=True) g = p.add_mutually_exclusive_group(required=True) g.add_argument("--years", type=float) g.add_argument("--months", type=int) p.add_argument("--start") p.add_argument("--extra", type=float, default=0.0) p.add_argument("--extra-from", type=int, default=1) p.add_argument("--lump", action="append", default=[]) p.add_argument("--payment", type=float) p.add_argument("--fee", type=float, default=0.0) p.add_argument("--schedule") p.add_argument("--yearly", action="store_true") p.add_argument("--json", action="store_true") try: a = p.parse_args(argv) except SystemExit as e: return 0 if e.code == 0 else 2 try: n = a.months if a.months else round(a.years * 12) if a.principal <= 0 or n <= 0 or a.rate < 0 or a.extra < 0 or a.fee < 0: raise ValueError("principal and term must be positive; rate, extra, and fee cannot be negative") if a.rate > 100: raise ValueError("rate is in percent per year, e.g. 4.2 for 4.2%") lumps = {} for item in a.lump: num, _, amt = item.partition(":") try: if not num.isdigit(): raise ValueError value = float(amt) except ValueError: raise ValueError(f"--lump must look like 24:5000 (payment number:amount), got {item!r}") from None lumps[int(num)] = lumps.get(int(num), 0.0) + value if a.start: try: y, m = a.start.split("-") start = (int(y), int(m)) except ValueError: raise ValueError(f"--start must look like 2026-11, got {a.start!r}") from None if not 1 <= start[1] <= 12: raise ValueError("--start month must be 01 to 12") else: t = dt.date.today() start = add_months((t.year, t.month), 1) base_pay, base_rows = schedule(a.principal, a.rate, n, payment=None) scenario = bool(a.extra or lumps or a.payment) pay, rows = schedule(a.principal, a.rate, n, payment=a.payment, extra=a.extra, extra_from=a.extra_from, lumps=lumps) if scenario else (base_pay, base_rows) except ValueError as e: print(f"error: {e}", file=sys.stderr) return 2 base_sum, plan_sum = summarize(base_rows), summarize(rows) result = { "inputs": {"principal": a.principal, "annual_rate_pct": a.rate, "term_months": n, "first_payment": fmt_ym(start), "extra_monthly": a.extra, "extra_from_payment": a.extra_from, "lumps": lumps, "fixed_payment": a.payment, "fees": a.fee}, "baseline": {"monthly_payment": base_pay, **base_sum, "payoff": fmt_ym(add_months(start, base_sum["months"] - 1))}, } if a.fee: result["baseline"]["apr_estimate_pct"] = apr_estimate(a.principal, a.fee, base_pay, n) result["baseline"]["total_cost_incl_fees"] = round(base_sum["total_interest"] + a.fee, 2) if scenario: result["plan"] = {"monthly_payment": pay, **plan_sum, "payoff": fmt_ym(add_months(start, plan_sum["months"] - 1)), "months_saved": base_sum["months"] - plan_sum["months"], "interest_saved": round(base_sum["total_interest"] - plan_sum["total_interest"], 2)} if plan_sum["total_extra"]: result["plan"]["interest_saved_per_100_extra"] = round(100 * result["plan"]["interest_saved"] / plan_sum["total_extra"], 2) yearly = [] for i in range(0, len(rows), 12): chunk = rows[i:i + 12] yearly.append({"year": i // 12 + 1, "paid": round(sum(x["payment"] + x["extra"] for x in chunk), 2), "interest": round(sum(x["interest"] for x in chunk), 2), "principal": round(sum(x["principal"] for x in chunk), 2), "end_balance": chunk[-1]["balance"]}) result["yearly"] = yearly first = base_rows[0] result["baseline"]["first_payment_split"] = {"interest": first["interest"], "principal": first["principal"]} if a.schedule: try: with open(a.schedule, "w", newline="", encoding="utf-8") as f: w = csv.writer(f) w.writerow(["n", "month", "payment", "extra", "interest", "principal", "balance"]) for x in rows: w.writerow([x["n"], fmt_ym(add_months(start, x["n"] - 1)), f"{x['payment']:.2f}", f"{x['extra']:.2f}", f"{x['interest']:.2f}", f"{x['principal']:.2f}", f"{x['balance']:.2f}"]) except OSError as e: print(f"error: cannot write schedule {a.schedule}: {e.strerror}", file=sys.stderr) return 2 if a.json: print(json.dumps(result, indent=2)) return 0 b = result["baseline"] print(f"Loan {a.principal:,.2f} at {a.rate}% for {n} months, first payment {fmt_ym(start)}") print(f"Baseline: {b['monthly_payment']:,.2f} per month, interest {b['total_interest']:,.2f}, " f"total paid {b['total_paid']:,.2f}, paid off {b['payoff']} ({b['months']} payments)") if a.fee: print(f" With fees {a.fee:,.2f}: total cost of borrowing {b['total_cost_incl_fees']:,.2f}, APR about {b['apr_estimate_pct']}%") print(f" First payment: {first['interest']:,.2f} interest, {first['principal']:,.2f} principal") if scenario: pl = result["plan"] what = [] if a.payment: what.append(f"fixed payment {a.payment:,.2f}") if a.extra: what.append(f"+{a.extra:,.2f}/month" + (f" from payment {a.extra_from}" if a.extra_from > 1 else "")) if lumps: what.append("lump sums " + ", ".join(f"{v:,.2f} in payment {k}" for k, v in sorted(lumps.items()))) print(f"Plan ({'; '.join(what)}): paid off {pl['payoff']} ({pl['months']} payments), " f"interest {pl['total_interest']:,.2f}") ms = pl["months_saved"] if ms >= 0 and pl["interest_saved"] >= 0: print(f" Saves {pl['interest_saved']:,.2f} interest and {ms} months ({ms // 12} y {ms % 12} m)" + (f"; every 100 of extra saves {pl['interest_saved_per_100_extra']:,.2f}" if pl.get("interest_saved_per_100_extra") else "")) else: print(f" WARNING: this plan costs {-pl['interest_saved']:,.2f} MORE interest and takes {-ms} months longer than the baseline") if a.yearly: print(f"\nYear-by-year ({'plan' if scenario else 'baseline'}):") print(f"{'Year':>4} {'Paid':>12} {'Interest':>12} {'Principal':>12} {'Balance':>12}") for y in yearly: print(f"{y['year']:>4} {y['paid']:>12,.2f} {y['interest']:>12,.2f} {y['principal']:>12,.2f} {y['end_balance']:>12,.2f}") if a.schedule: print(f"\nSchedule written to {a.schedule} ({len(rows)} rows)") return 0 if __name__ == "__main__": sys.exit(main(sys.argv[1:]))
【中文说明】用于审查 Dockerfile 的安全性、镜像体积、构建缓存和运行可靠性，并提出修改版本及忽略文件建议。粘贴 Dockerfile，补充应用类型与构建环境，即可排查高权限运行、密钥泄露、缓存失效和依赖安装等问题。 Reviews Dockerfiles for security, image size, build cache, and runtime reliability: secrets baked into layers, running as root, unpinned base images, curl piped to a shell, ADD misuse, leftover apt and pip caches, cache-busting dependency installs, shell-form CMD, and missing multi-stage builds or health checks, then writes a corrected Dockerfile and .dockerignore. Use when a user shares a Dockerfile and asks "review my Dockerfile", "why is my image so big?", or "is this container secure?".
---
name: dockerfile-best-practices-reviewer
description: Reviews Dockerfiles for security, image size, build cache, and runtime reliability: secrets baked into layers, running as root, unpinned base images, curl piped to a shell, ADD misuse, apt and pip caches left in layers, dependency installs that bust the cache, shell-form CMD, and missing multi-stage builds or health checks, then writes a corrected Dockerfile and .dockerignore. Use when a user shares a Dockerfile and asks "review my Dockerfile", "why is my image so big?", "is this container secure?", or "why does every build reinstall dependencies?". Includes a tested stdlib Python linter.
---
# Dockerfile Best Practices Reviewer
You help developers ship container images that are small, fast to rebuild, and safe to run. You read a Dockerfile the way a careful platform engineer would: you find what leaks secrets or gives away root, what makes the image heavy, and what forces slow rebuilds, and you return a corrected Dockerfile that still builds and runs the same app.
## Files in this skill
- `scripts/lint_dockerfile.py` - parses a Dockerfile (continuation lines, multi-stage builds) and reports HIGH, WARN, and INFO findings with a fix for each (Python 3 standard library only)
- `references/dockerfile-rules.md` - every rule the linter checks, why it matters, and the correct pattern
- `references/language-patterns.md` - proven multi-stage patterns for Node.js, Python, Go, and Java, plus a starter .dockerignore
- `templates/dockerfile-review.md` - the review report to return
- `examples/example-node-api-review.md` - a worked review of a Node.js API Dockerfile
## Workflow
### 1. Collect the facts
Ask, or assume and say so:
- The Dockerfile, and the .dockerignore if there is one.
- Language and framework, how the app is built and started, and the port.
- Where it runs (Kubernetes, ECS, a VM with Docker Compose, a laptop) and whether that platform already has health checks.
- Constraints: required base images (company registry, distroless, Alpine or Debian), native dependencies, and whether the build uses BuildKit.
### 2. Run the linter
```bash
python3 scripts/lint_dockerfile.py Dockerfile
python3 scripts/lint_dockerfile.py services/*/Dockerfile --fail-on warn # stricter, for CI
python3 scripts/lint_dockerfile.py - --json < Dockerfile
```
The linter is a fast first pass based on patterns, not a full build. Read the whole Dockerfile yourself as well: it cannot know whether a package is really needed, whether a file copied in contains secrets, or whether the app writes to a path that a non-root user cannot access. If you cannot run the script, apply `references/dockerfile-rules.md` by hand.
### 3. Fix in this order
1. Security (HIGH): remove literal secrets from ENV, ARG, and RUN and use BuildKit secret mounts or runtime configuration instead; tell the user to rotate any secret that was ever committed or pushed in an image. Add a non-root USER. Replace curl-pipe-shell and chmod 777.
2. Reproducibility: pin base images to a version tag (digest for high-assurance builds) and use lockfile installs (npm ci, pip with a requirements lock, go mod download).
3. Build cache: copy dependency manifests first, install, then copy the source. Combine update and install in one RUN.
4. Size: multi-stage build, slim or distroless runtime base, no recommended packages, clean package caches in the same layer, a good .dockerignore.
5. Runtime: exec-form CMD or ENTRYPOINT, one process per container, a HEALTHCHECK if the platform has no probe, sensible WORKDIR and file ownership.
Rerun the linter on the corrected file until there are no HIGH findings and every WARN is fixed or explained.
### 4. Deliver
Fill in `templates/dockerfile-review.md`: summary, findings table, the corrected Dockerfile with short comments, a .dockerignore, how to build and test it, and an estimate of the size and rebuild-time impact. Use patterns from `references/language-patterns.md` and follow the style of `examples/example-node-api-review.md`.
## Rules
- Keep the app's behavior: same start command, port, environment variables, and files at runtime, unless the user agrees to change them.
- Never print or repeat a real secret found in the file; refer to it by name and tell the user to rotate it.
- Do not claim exact image sizes without a build; give ranges and say how to measure (`docker image ls`, `docker history`).
- Prefer official, maintained base images; mention the trade-offs of Alpine (musl) versus Debian slim versus distroless for the user's language.
- Explain each change in one line so the user learns the rule, not just the fix.
FILE:references/dockerfile-rules.md
# Dockerfile rules checked by the linter
Each rule: what is detected, why it matters, and the pattern to use instead.
## HIGH (security)
### secret-in-env / secret-in-run
Detected: ENV or ARG with a name like PASSWORD, SECRET, TOKEN, API_KEY, ACCESS_KEY, PRIVATE_KEY, CREDENTIALS and a literal value; `--password=...` or `user:pass@` URLs in RUN; ARG names that look like secrets (WARN).
Why: every ENV value and build arg is stored in the image config and history (`docker history --no-trunc`). Anyone who can pull the image can read it. Deleting it in a later layer does not remove it.
Use instead:
```dockerfile
# syntax=docker/dockerfile:1
RUN --mount=type=secret,id=npm_token \
NPM_TOKEN="$(cat /run/secrets/npm_token)" npm ci
```
Build with `docker build --secret id=npm_token,src=$HOME/.npm_token .`. Pass runtime secrets as environment variables from the platform or a secrets manager. Rotate any secret that was ever in an image.
### root-user
Detected: no USER in the final stage, or USER root / 0.
Why: a process that escapes the app as root inside the container has far more power over the host and other containers.
Use instead: create a user and switch before CMD; many official images already include one (`node`, `nobody`).
```dockerfile
RUN groupadd -r app && useradd -r -g app -u 10001 app
COPY --chown=app:app . /app
USER app
```
### curl-pipe-shell
Detected: `curl ... | sh` or `wget ... | bash`.
Why: runs whatever the server returns, with no integrity check; a compromised or changed script ends up in every build.
Use instead: download, verify a checksum, then run.
```dockerfile
RUN curl -fsSLo /tmp/install.sh https://example.com/install.sh \
&& echo "<sha256> /tmp/install.sh" | sha256sum -c - \
&& sh /tmp/install.sh && rm /tmp/install.sh
```
### chmod-777
Detected: `chmod 777` or `chmod -R 777`.
Why: any user or process in the container can modify the files, including app code.
Use instead: `COPY --chown=app:app` and `chmod 755` for executables, `644` for files.
## WARN
| Rule | Detected | Why | Fix |
|---|---|---|---|
| unpinned-base | FROM without tag or with :latest | builds change silently; hard to reproduce or roll back | `FROM node:20.11-bookworm-slim` or pin a digest `@sha256:...` |
| use-copy | ADD for local files or URLs | ADD auto-extracts archives and downloads without checksums | COPY for files; curl with checksum or `ADD --checksum=` for URLs; ADD is fine for local tar archives you want extracted |
| update-alone | `apt-get update` (or apk/yum) without install in the same RUN | the cached update layer gets reused with a newer install line, so packages come from a stale index | `RUN apt-get update && apt-get install -y ...` |
| cache-bust | dependency install after `COPY . .` | any source change invalidates the install layer, so every build reinstalls everything | copy manifests first, install, then copy the source |
| sudo | sudo in RUN | not needed (build runs as root until USER), adds a setuid binary | run the step before USER |
| ssh-port | EXPOSE 22 | containers should not run SSH daemons | `docker exec`, `kubectl exec` |
| multiple-cmd | more than one CMD or ENTRYPOINT in a stage | only the last one counts; usually a mistake | keep one |
## INFO
| Rule | Fix |
|---|---|
| apt-recommends | `apt-get install -y --no-install-recommends ...` |
| apt-lists | end the same RUN with `&& rm -rf /var/lib/apt/lists/*` |
| apk-cache | `apk add --no-cache ...` |
| pip-cache | `pip install --no-cache-dir ...` or a BuildKit cache mount |
| npm-ci | `npm ci --omit=dev` uses the lockfile exactly |
| shell-form | `CMD ["node", "server.js"]` so the app is PID 1 and receives SIGTERM for graceful shutdown |
| cd-in-run | `WORKDIR /app` |
| no-healthcheck | `HEALTHCHECK CMD curl -fsS http://localhost:3000/health || exit 1` if the platform has no probe; Kubernetes ignores HEALTHCHECK and uses its own probes |
| single-stage | build in a builder stage; copy only the output into a slim runtime stage |
| many-layers | combine related RUN steps; fewer, purposeful layers |
| dockerignore | exclude `.git`, `node_modules`, `.env*`, build output, logs, and local secrets |
## What the linter cannot see
- Whether copied files contain secrets (check .dockerignore and the repo).
- Vulnerable packages in the base image or dependencies: suggest an image scanner in CI.
- Whether the non-root user can write where the app writes (logs, uploads, caches).
- Architecture issues (arm64 vs amd64), native modules, and Alpine musl compatibility.
FILE:references/language-patterns.md
# Multi-stage patterns by language
Adapt versions to the project. All examples run as a non-root user, use exec-form CMD, and keep build tools out of the runtime image.
## Node.js (npm)
```dockerfile
# syntax=docker/dockerfile:1
FROM node:20.11-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
FROM node:20.11-bookworm-slim AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
FROM node:20.11-bookworm-slim
ENV NODE_ENV=production
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build --chown=node:node /app/dist ./dist
USER node
EXPOSE 3000
CMD ["node", "dist/server.js"]
```
Skip the build stage if there is no compile step. Use `node:<version>-alpine` only if native modules work with musl.
## Python (pip)
```dockerfile
FROM python:3.12-slim AS build
WORKDIR /app
RUN python -m venv /venv
ENV PATH="/venv/bin:$PATH"
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
FROM python:3.12-slim
ENV PATH="/venv/bin:$PATH" PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
WORKDIR /app
COPY --from=build /venv /venv
RUN useradd -r -u 10001 app
COPY --chown=app:app . .
USER app
EXPOSE 8000
CMD ["gunicorn", "-b", "0.0.0.0:8000", "app:app"]
```
If packages need compilers (psycopg2, numpy from source), install `build-essential` only in the build stage.
## Go
```dockerfile
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/app
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
USER nonroot
ENTRYPOINT ["/app"]
```
A static Go binary on distroless or scratch gives images of a few MB to tens of MB.
## Java (Maven)
```dockerfile
FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /src
COPY pom.xml .
RUN mvn -q dependency:go-offline
COPY src ./src
RUN mvn -q package -DskipTests
FROM eclipse-temurin:21-jre
RUN useradd -r -u 10001 app
WORKDIR /app
COPY --from=build --chown=app:app /src/target/app.jar app.jar
USER app
EXPOSE 8080
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75", "-jar", "app.jar"]
```
## Starter .dockerignore
```
.git
.gitignore
.dockerignore
Dockerfile*
node_modules
dist
build
target
__pycache__
*.pyc
.venv
.env
.env.*
*.log
coverage
.idea
.vscode
secrets/
*.pem
*.key
```
## Measuring the result
- `docker build -t app:review .` then `docker image ls app:review` for size.
- `docker history app:review` to see which layers are large.
- Change one source file and rebuild: the dependency install step should say CACHED.
FILE:templates/dockerfile-review.md
# Dockerfile review: service_name
**App:** language_and_framework, started with start_command, port port
**Runs on:** platform
**Linter:** HIGH high / WARN warn / INFO info before; HIGH 0 / WARN warn_after / INFO info_after after
## Summary
Two or three sentences: the biggest security problem, the biggest size or speed problem, and the expected improvement.
## Findings
| # | Level | Line | Problem | Fix |
|---|---|---|---|---|
| 1 | HIGH | | | |
## Action needed outside the Dockerfile
- Rotate: (names of any secrets that were in the file or image, never their values)
- Platform: health probes, read-only filesystem, resource limits
## Corrected Dockerfile
```dockerfile
(full corrected Dockerfile with one-line comments on changed parts)
```
## .dockerignore
```
(entries)
```
## Build and verify
```bash
docker build -t image:review .
docker run --rm -p port:port image:review
docker image ls image:review
python3 scripts/lint_dockerfile.py Dockerfile
```
## Expected impact (estimate)
- Image size: from about ... to about ...
- Rebuild after a source change: dependency layer cached
- Security: non-root, no secrets in layers, pinned base
FILE:examples/example-node-api-review.md
# Example: review of a Node.js API Dockerfile
## User request
"Our orders API image is 1.2 GB and every build reinstalls all npm packages, even when I change one line. Can you review the Dockerfile? It runs on Kubernetes, Express app, `node server.js`, port 3000. No native modules as far as I know."
## Input Dockerfile
```dockerfile
FROM node
ENV NODE_ENV=production
ENV DB_PASSWORD=supersecret123
RUN apt-get update
RUN apt-get install -y curl git build-essential
WORKDIR /app
COPY . .
RUN npm install
RUN curl -fsSL https://example.com/install-agent.sh | bash
RUN chmod -R 777 /app
ADD config.json /app/config.json
EXPOSE 3000 22
CMD node server.js
```
## Linter run
```
$ python3 scripts/lint_dockerfile.py Dockerfile
Dockerfile: 1 stage(s): node
HIGH 4 WARN 5 INFO 7
HIGH line 1 root-user the final image runs as root
HIGH line 3 secret-in-env ENV DB_PASSWORD contains a literal secret; it is stored in the image layers and history
HIGH line 9 curl-pipe-shell downloads a script and pipes it straight into a shell
HIGH line 10 chmod-777 chmod 777 makes files writable by every user
WARN line 1 unpinned-base base image 'node' has no tag, so it means :latest and changes without notice
WARN line 4 update-alone 'apt-get update' runs in its own layer; a later install may use a stale cached index
WARN line 8 cache-bust dependencies are installed after 'COPY . .' (line 7), so any source change reinstalls them
WARN line 11 use-copy ADD is used for local files; it also auto-extracts archives and fetches URLs
WARN line 12 ssh-port port 22 (SSH) is exposed; containers should not run an SSH server
INFO line 1 no-healthcheck no HEALTHCHECK in the final stage
INFO line 1 single-stage build tools are installed in the only stage, so they ship in the final image
INFO line 5 apt-recommends apt installs recommended packages too
INFO line 5 apt-lists apt package lists are left in the layer (tens of MB)
INFO line 7 dockerignore COPY of the whole build context
INFO line 8 npm-ci npm install can change the lockfile and versions
INFO line 13 shell-form shell-form CMD runs under /bin/sh -c, so the app is not PID 1 and may miss SIGTERM
```
## Review (filled template, shortened)
**Linter:** HIGH 4 / WARN 5 / INFO 7 before; HIGH 0 / WARN 0 / INFO 0 after
**Summary:** The image ships a database password in its layers and runs as root, and it pipes a remote script into bash. It is large because it uses the full `node` image with compilers and git, and slow because `COPY . .` comes before `npm install`. The corrected two-stage Dockerfile removes the secret, runs as the built-in `node` user, and caches dependencies; expect roughly 200 to 300 MB instead of 1.2 GB (measure with `docker image ls`).
| # | Level | Line | Problem | Fix |
|---|---|---|---|---|
| 1 | HIGH | 3 | DB_PASSWORD literal in ENV, visible in `docker history` | removed; inject from a Kubernetes Secret at runtime |
| 2 | HIGH | - | runs as root | `USER node` (built into the official image) |
| 3 | HIGH | 9 | remote install script piped into bash | removed: the agent now runs as a Kubernetes sidecar; if it must stay, download, verify the checksum, then run |
| 4 | HIGH | 10 | chmod -R 777 /app | `COPY --chown=node:node` instead |
| 5 | WARN | 1 | untagged `node` base | `node:20.11-bookworm-slim` |
| 6 | WARN | 4 | apt-get update in its own layer | combined with install and list cleanup |
| 7 | WARN | 8 | npm install after `COPY . .` | manifests first, `npm ci --omit=dev` in a deps stage |
| 8 | WARN | 11 | ADD for a local file | COPY |
| 9 | WARN | 12 | EXPOSE 22 | removed; use `kubectl exec` |
| 10 | INFO | 5, 13 | build tools and git in the runtime image, shell-form CMD | dropped (no native modules), exec-form CMD |
**Action needed outside the Dockerfile:** rotate DB_PASSWORD now (it is in every image pushed so far), and add it to the Deployment from a Kubernetes Secret. Kubernetes ignores HEALTHCHECK, so also add a readiness and liveness probe on /health; the HEALTHCHECK below helps local Docker and Compose.
## Corrected Dockerfile
```dockerfile
# syntax=docker/dockerfile:1
FROM node:20.11-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
FROM node:20.11-bookworm-slim
ENV NODE_ENV=production
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl \
&& rm -rf /var/lib/apt/lists/*
COPY --from=deps /app/node_modules ./node_modules
COPY --chown=node:node src/ ./src/
COPY --chown=node:node config.json ./
USER node
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=3s CMD curl -fsS http://localhost:3000/health || exit 1
CMD ["node", "src/server.js"]
```
## .dockerignore
```
.git
node_modules
.env
.env.*
*.log
coverage
Dockerfile*
```
## Verify
```
$ python3 scripts/lint_dockerfile.py Dockerfile
Dockerfile: 2 stage(s): node:20.11-bookworm-slim AS deps, node:20.11-bookworm-slim
HIGH 0 WARN 0 INFO 0
no problems found
```
Note: the source now lives in `src/`, so the start command became `node src/server.js`; if your entry file is elsewhere, keep your original path. If a dependency turns out to need compilers, install `build-essential` and `python3` in the deps stage only.
FILE:scripts/lint_dockerfile.py
#!/usr/bin/env python3
"""Review a Dockerfile for security, image size, build cache, and reliability problems.
Usage:
python3 lint_dockerfile.py Dockerfile [more Dockerfiles] [--json] [--fail-on high|warn|info]
python3 lint_dockerfile.py - < Dockerfile
Checks (rule ids in brackets):
HIGH secrets in ENV/ARG or in RUN (secret-in-env, secret-in-run), final stage runs as root
(root-user), pipe from curl/wget to a shell (curl-pipe-shell), chmod 777 (chmod-777)
WARN untagged or :latest base image (unpinned-base), ADD for local files or URLs (use-copy),
apt/apk/yum update without install in the same RUN (update-alone), COPY of the whole
context before installing dependencies (cache-bust), sudo (sudo), EXPOSE 22 (ssh-port),
more than one CMD or ENTRYPOINT in a stage (multiple-cmd)
INFO apt-get without --no-install-recommends or without cleaning lists, pip without
--no-cache-dir, npm install instead of npm ci, shell-form CMD/ENTRYPOINT, cd in RUN
instead of WORKDIR, no HEALTHCHECK, single-stage build with build tools, many RUN layers,
COPY of the whole context (check .dockerignore)
Exit codes: 0 ok, 1 findings at or above --fail-on (default high), 2 usage or read error.
Python 3 standard library only.
"""
import json
import re
import shlex
import sys
LEVELS = {"info": 0, "warn": 1, "high": 2}
SECRET_KEY = re.compile(r"(PASSWORD|PASSWD|SECRET|TOKEN|API_?KEY|ACCESS_?KEY|PRIVATE_?KEY|CREDENTIALS?)", re.I)
BUILD_TOOLS = re.compile(r"\b(build-essential|gcc|g\+\+|make|cmake|maven|gradle|golang|rustc|cargo|npm run build|go build|mvn )", re.I)
def logical_lines(text):
"""Join continuation lines; return list of (line_no, instruction, args). Skips comments and parser directives."""
out, buf, start = [], "", None
escape = "\\"
for i, raw in enumerate(text.replace("\r\n", "\n").split("\n"), 1):
m = re.match(r"^#\s*escape\s*=\s*(\S)", raw)
if m and not out and not buf:
escape = m.group(1)
continue
s = raw.strip()
if not buf and (not s or s.startswith("#")):
continue
if buf and s.startswith("#"):
continue
if start is None:
start = i
if s.endswith(escape):
buf += s[:-1] + " "
continue
buf += s
parts = buf.split(None, 1)
out.append((start, parts[0].upper(), parts[1] if len(parts) > 1 else ""))
buf, start = "", None
if buf:
parts = buf.split(None, 1)
out.append((start, parts[0].upper(), parts[1] if len(parts) > 1 else ""))
return out
def review(text):
lines = logical_lines(text)
findings = []
def add(level, rule, line, msg, fix):
findings.append({"level": level, "rule": rule, "line": line, "message": msg, "fix": fix})
if not any(ins == "FROM" for _, ins, _ in lines):
raise ValueError("no FROM instruction found (is this a Dockerfile?)")
stages, cur = [], None
for ln, ins, args in lines:
if ins == "FROM":
m = re.match(r"(?:--platform=\S+\s+)?(\S+)(?:\s+AS\s+(\S+))?", args, re.I)
cur = {"line": ln, "image": m.group(1) if m else args, "name": m.group(2) if m else None, "ins": []}
stages.append(cur)
elif cur is not None:
cur["ins"].append((ln, ins, args))
names = {s["name"].lower() for s in stages if s["name"]}
arg_names = {a.split("=")[0].strip() for _, i, a in lines if i == "ARG"}
for s in stages:
img = s["image"]
if img.lower() in names or img == "scratch" or img.startswith("$"):
pass
elif "@sha256:" in img:
pass
elif ":" not in img.split("/")[-1]:
add("warn", "unpinned-base", s["line"], f"base image '{img}' has no tag, so it means :latest and changes without notice",
f"pin a version, e.g. {img}:<version>-slim, or a digest")
elif img.endswith(":latest"):
add("warn", "unpinned-base", s["line"], f"base image '{img}' uses :latest", "pin a specific version tag or digest")
final = stages[-1]
run_count = 0
for idx, s in enumerate(stages):
cmds = [x for x in s["ins"] if x[1] in ("CMD", "ENTRYPOINT")]
for kind in ("CMD", "ENTRYPOINT"):
k = [x for x in cmds if x[1] == kind]
if len(k) > 1:
add("warn", "multiple-cmd", k[-1][0], f"{len(k)} {kind} instructions in one stage; only the last one counts",
f"keep a single {kind}")
copied_all = None
for ln, ins, args in s["ins"]:
low = args.lower()
if ins in ("ENV", "ARG"):
pairs = re.findall(r"([A-Za-z_][A-Za-z0-9_]*)(?:(?:=|\s+)(\"[^\"]*\"|'[^']*'|[^\s=]\S*))?", args) if ins == "ARG" or "=" in args else [tuple((args.split(None, 1) + [""])[:2])]
for key, val in pairs:
if SECRET_KEY.search(key) and val and not val.startswith("$") and val.strip("\"'"):
add("high", "secret-in-env", ln, f"{ins} {key} contains a literal secret; it is stored in the image layers and history",
"pass secrets at build time with RUN --mount=type=secret, or at run time via environment or a secrets manager")
if ins == "ARG" and any(SECRET_KEY.search(k) for k, _ in pairs) and not any(v for k, v in pairs if SECRET_KEY.search(k)):
add("warn", "secret-in-env", ln, f"ARG {pairs[0][0]} looks like a secret; build args are visible in image history",
"use RUN --mount=type=secret instead of a build arg")
if ins == "RUN":
run_count += 1
if re.search(r"(curl|wget)\b[^|;&]*\|\s*(sudo\s+)?(ba|z|da)?sh\b", low):
add("high", "curl-pipe-shell", ln, "downloads a script and pipes it straight into a shell",
"download to a file, verify a checksum or signature, then run it")
if re.search(r"chmod\s+(-r\s+)?0?777\b", low):
add("high", "chmod-777", ln, "chmod 777 makes files writable by every user", "grant only the permissions needed, e.g. chmod 755 or chown to the app user")
if re.search(r"(--password[= ]\S+|://[^/\s:@]+:[^@\s$]+@)", args) or re.search(r"\b(token|api[_-]?key)=\w{8,}", low):
add("high", "secret-in-run", ln, "a password, token, or credentials URL appears in a RUN command and stays in the image history",
"use RUN --mount=type=secret or fetch credentials at run time")
if re.search(r"\bsudo\b", low):
add("warn", "sudo", ln, "sudo inside a Dockerfile is unnecessary and widens the attack surface", "run the step before switching USER, without sudo")
for upd, inst in (("apt-get update", "apt-get install"), ("apt update", "apt install"), ("apk update", "apk add"), ("yum update", "yum install")):
if upd in low and inst not in low:
add("warn", "update-alone", ln, f"'{upd}' runs in its own layer; a later install may use a stale cached index",
f"combine: RUN {upd} && {inst} ... in one RUN")
if "apt-get install" in low or "apt install" in low:
if "--no-install-recommends" not in low:
add("info", "apt-recommends", ln, "apt installs recommended packages too", "add --no-install-recommends")
if "/var/lib/apt/lists" not in low:
add("info", "apt-lists", ln, "apt package lists are left in the layer (tens of MB)", "end the same RUN with && rm -rf /var/lib/apt/lists/*")
if "apk add" in low and "--no-cache" not in low:
add("info", "apk-cache", ln, "apk cache is kept in the layer", "use apk add --no-cache")
if re.search(r"\bpip3?\s+install\b", low) and "--no-cache-dir" not in low and "--mount=type=cache" not in low:
add("info", "pip-cache", ln, "pip keeps its download cache in the layer", "add --no-cache-dir (or use a cache mount)")
if any(re.fullmatch(r"\s*npm\s+(install|i)(\s+--?(?!g\b|global\b)[\w-]+(=\S+)?)*\s*", seg) for seg in re.split(r"&&|;|\|\|", low)):
add("info", "npm-ci", ln, "npm install can change the lockfile and versions", "use npm ci (with --omit=dev for production)")
if re.match(r"cd\s+\S+\s*&&", low) or re.search(r"&&\s*cd\s+/", low):
add("info", "cd-in-run", ln, "cd in RUN only affects that one command", "use WORKDIR /path")
if copied_all and re.search(r"\b(npm (ci|install)|pip3? install -r|poetry install|bundle install|go mod download|composer install|yarn install)\b", low):
add("warn", "cache-bust", ln, f"dependencies are installed after 'COPY {copied_all[1]}' (line {copied_all[0]}), so any source change reinstalls them",
"copy only the manifest and lockfile first (e.g. COPY package*.json ./), install, then COPY the rest")
copied_all = None
if ins == "ADD":
src = args.split()[0] if args.split() else ""
if re.match(r"https?://", src):
add("warn", "use-copy", ln, "ADD with a URL downloads without checksum verification and leaves the file in a layer",
"use RUN curl -fsSL ... with a checksum check, or ADD --checksum=sha256:...")
elif not re.search(r"\.(tar|tar\.gz|tgz|tar\.xz|tar\.bz2)$", src):
add("warn", "use-copy", ln, "ADD is used for local files; it also auto-extracts archives and fetches URLs", "use COPY for plain files")
if ins == "COPY" and not args.startswith("--from"):
parts = [p for p in args.split() if not p.startswith("--")]
if parts and parts[0] in (".", "./", "*"):
copied_all = (ln, " ".join(parts))
add("info", "dockerignore", ln, "COPY of the whole build context", "make sure .dockerignore excludes .git, node_modules, .env, build output, and secrets")
if ins == "EXPOSE" and re.search(r"\b22\b", args):
add("warn", "ssh-port", ln, "port 22 (SSH) is exposed; containers should not run an SSH server", "use docker exec / kubectl exec for access")
if ins in ("CMD", "ENTRYPOINT") and not args.strip().startswith("["):
add("info", "shell-form", ln, f"shell-form {ins} runs under /bin/sh -c, so the app is not PID 1 and may miss SIGTERM",
f'use exec form: {ins} ["executable", "arg"]')
users = [(ln, a.strip()) for ln, i, a in final["ins"] if i == "USER"]
if not users or users[-1][1].split(":")[0] in ("root", "0"):
where = users[-1][0] if users else final["line"]
add("high", "root-user", where, "the final image runs as root",
"create an unprivileged user (e.g. RUN useradd -r -u 10001 app) and add USER app before CMD")
if not any(i == "HEALTHCHECK" for _, i, _ in final["ins"]):
add("info", "no-healthcheck", final["line"], "no HEALTHCHECK in the final stage", "add HEALTHCHECK CMD ... if no orchestrator health probe is configured")
if len(stages) == 1 and any(BUILD_TOOLS.search(a) for _, i, a in final["ins"] if i == "RUN"):
add("info", "single-stage", final["line"], "build tools are installed in the only stage, so they ship in the final image",
"use a multi-stage build: compile in a builder stage, COPY --from=builder only the output")
if run_count > 10:
add("info", "many-layers", final["line"], f"{run_count} RUN instructions; related steps can be combined", "group related commands with && in fewer RUN steps")
return {"stages": [{"line": s["line"], "image": s["image"], "name": s["name"]} for s in stages], "findings": findings}
def main(argv):
as_json, fail_on, files = False, "high", []
it = iter(argv)
for a in it:
if a == "--json":
as_json = True
elif a == "--fail-on":
fail_on = next(it, "").lower()
if fail_on not in LEVELS:
print("error: --fail-on must be high, warn, or info", file=sys.stderr)
return 2
elif a in ("-h", "--help"):
print(__doc__)
return 0
elif a.startswith("--"):
print(f"error: unknown option {a}", file=sys.stderr)
return 2
else:
files.append(a)
if not files:
print("error: give a Dockerfile path, or - for stdin", file=sys.stderr)
return 2
reports, worst = [], -1
for f in files:
try:
text = sys.stdin.read() if f == "-" else open(f, encoding="utf-8").read()
r = review(text)
except (OSError, UnicodeDecodeError, ValueError) as e:
print(f"error: {f}: {e}", file=sys.stderr)
return 2
r["file"] = f
r["findings"].sort(key=lambda x: (-LEVELS[x["level"]], x["line"]))
for x in r["findings"]:
worst = max(worst, LEVELS[x["level"]])
reports.append(r)
if as_json:
print(json.dumps(reports, indent=2))
else:
for r in reports:
c = {lv: sum(1 for x in r["findings"] if x["level"] == lv) for lv in LEVELS}
st = ", ".join(f"{s['image']}" + (f" AS {s['name']}" if s["name"] else "") for s in r["stages"])
print(f"{r['file']}: {len(r['stages'])} stage(s): {st}")
print(f" HIGH {c['high']} WARN {c['warn']} INFO {c['info']}")
for x in r["findings"]:
print(f" {x['level'].upper():<4} line {x['line']:<3} {x['rule']:<15} {x['message']}")
print(f" fix: {x['fix']}")
if not r["findings"]:
print(" no problems found")
return 1 if worst >= LEVELS[fail_on] else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))【中文说明】用于逐 token 解释 Python 正则表达式，测试必须匹配和不得匹配的样例，并检查灾难性回溯、未转义点号、空匹配、缺少锚点等风险。使用时提供正则、目标语言、正反例和期望捕获组，模型会展示匹配跨度与命名组，并在必要时给出更安全的改写。 Explains Python regular expressions token by token, tests them on must-match and must-not-match samples with spans and named groups, and flags risky patterns: nested quantifiers that backtrack catastrophically, [A-z], unescaped dots, empty matches, and missing anchors, then proposes a safer rewrite. Use when a user asks "what does this regex do?", "why doesn't it match?", or "is this regex safe?". Includes a tested stdlib Python checker.
--- name: regex-pattern-explainer-and-tester description: Explains Python regular expressions token by token, tests them on must-match and must-not-match samples with spans and named groups, and flags risky patterns: nested quantifiers that backtrack catastrophically, [A-z], unescaped dots, empty matches, and missing anchors, then proposes a safer rewrite. Use when a user asks "what does this regex do?", "why doesn't it match?", or "is this regex safe?". Includes a tested stdlib Python checker. --- # Regex Pattern Explainer and Tester You help people understand, fix, and trust regular expressions. You explain a pattern in plain words, prove what it does with real test cases, and find the traps that make regexes fail in production: inputs that should not pass, inputs that should pass but do not, and patterns that hang on long input. You return a corrected pattern with a test table, not just an opinion. ## Files in this skill - `scripts/regex_check.py` - explains a pattern from Python's own regex parser, runs sample inputs with "+" (must match) and "-" (must not match) expectations, and reports HIGH, WARN, and INFO risks; `--probe` times the pattern on generated worst-case input in a subprocess (Python 3 standard library only) - `references/regex-pitfalls.md` - every risk the checker reports, why it matters, and the safe pattern to use instead - `references/flavor-differences.md` - how Python `re` differs from JavaScript, PCRE, Go RE2, .NET, Java, and POSIX grep, so a fix works where the user runs it - `templates/regex-review.md` - the review report to return - `examples/example-email-validation-review.md` - a worked review of an email check that hangs on long input ## Workflow ### 1. Collect the facts Ask, or assume and say so: - The pattern exactly as written in the code (raw string or escaped string literal) and the language or tool that runs it. - What it must accept and what it must reject: 4 to 10 real examples of each, including edge cases (empty input, spaces, Unicode, very long input). - How it is used: validate a whole value (fullmatch), find the first match (search), extract all matches (findall), or replace. - Where the input comes from. Untrusted input (web forms, logs, API payloads) makes catastrophic backtracking a security issue. ### 2. Run the checker ```bash python3 scripts/regex_check.py 'PATTERN' -t 'sample one' -t 'sample two' python3 scripts/regex_check.py 'PATTERN' --cases cases.txt --mode fullmatch --flags a python3 scripts/regex_check.py 'PATTERN' --probe --no-explain # confirm a backtracking risk python3 scripts/regex_check.py 'PATTERN' --cases - --json < cases.txt ``` A cases file has one sample per line: `+ text` must match, `- text` must not match, `# ...` is a comment. Always quote the pattern in single quotes in the shell. The checker uses Python's `re`; for other flavors, read `references/flavor-differences.md` and translate the result. If you cannot run the script, explain the pattern by hand with the same structure and apply `references/regex-pitfalls.md`. ### 3. Explain, then fix 1. Explain the pattern top to bottom in plain words, one line per token or group, using the checker's explanation as the base. 2. Show the test table: each sample, expected result, actual result, and the matched text or groups. 3. Fix in this order: patterns that can hang (HIGH), wrong accepts and wrong rejects (failed expectations), then clarity and the INFO notes. 4. Prefer simple, readable rewrites: one repeat instead of nested ones, negated classes like `[^,]*` instead of `.*`, explicit anchors `\A...\Z` or `fullmatch`, and `re.VERBOSE` with comments for long patterns. 5. Rerun the checker on the corrected pattern with the same cases plus any new edge cases until every expectation passes and there is no HIGH risk. Run `--probe` on the fix if the original had a HIGH risk. ### 4. Deliver Fill in `templates/regex-review.md`: the plain-language explanation, the test table before and after, the risks found, the corrected pattern in the user's language with correct escaping, and a short list of test cases to keep in their test suite. Follow the style of `examples/example-email-validation-review.md`. ## Rules - Never claim a pattern is correct without showing it on test cases; add the edge cases the user forgot. - Do not over-validate: for emails, URLs, and names, recommend a simple shape check plus a real verification step (sending an email, parsing the URL) instead of a giant pattern. - Keep the user's flavor: do not suggest lookbehind, possessive quantifiers, or atomic groups where their engine does not support them. - Escape correctly for the target language (raw strings in Python, double backslashes in Java and C# strings, slashes in JavaScript literals). - Treat catastrophic backtracking on untrusted input as a security bug (ReDoS) and say so plainly; suggest input length limits as a second defense. FILE:references/regex-pitfalls.md # Regex pitfalls checked by the script Each rule: what is detected, why it matters, and the pattern to use instead. Levels: HIGH can hang or be exploited, WARN usually accepts or rejects the wrong input, INFO is a habit worth fixing. ## HIGH (catastrophic backtracking, ReDoS) ### nested-quantifier Detected: a repeat inside another repeat where nothing else in the outer repeat is required, such as `(a+)+`, `(\w+\s?)+`, `([a-z0-9]+\.?)+`, `(\w+|\d+)+`. Why: on input that almost matches (a long run of letters followed by a char the pattern rejects), the engine tries every way to split the run between the inner and outer repeat: 2^n ways for n chars. 30 chars can take minutes. On a web form or API this is a denial-of-service bug. Use instead: one repeat that describes the whole run, with the separator required. ``` (\w+\s?)+$ -> \w+(?:\s\w+)*$ ([a-z0-9]+\.?)+@ -> [a-z0-9]+(?:\.[a-z0-9]+)*@ ``` Python 3.11+ also supports atomic groups `(?>...)` and possessive quantifiers `a++`, which never give back what they matched. ### ambiguous-repeat Detected: a repeat whose body has variable length and whose required parts can also be eaten by the variable part, such as `(x+x+)+`, `(aa|a)+`, `(foo|f\w+)+`. Why: same as above, the iterations can be cut in many ways. Growth may be slower (Fibonacci instead of 2^n) but is still exponential. Use instead: make each iteration start with a char nothing else in the body can match, or merge the parts into one class. Confirm both with `--probe`: the script runs the pattern on `c * n + tail` for n = 12 to 28 in a subprocess with a 2 second limit and reports how fast the time grows. A growth of more than 3x per 4 extra chars, or a timeout, confirms the risk. ## WARN ### nested-overlap A repeat whose body has a separator that the inner part also accepts, such as `([\w.]+\.)+`. Make the inner class exclude the separator: `(?:[\w-]+\.)+`. ### adjacent-overlap Two unbounded single-char repeats in a row that accept the same chars, such as `\d+\d+` or `.*.*`. The split point is ambiguous and a failing match backtracks polynomially (n^2, n^3). Keep one of them or make the boundary explicit. ### range-A-z `[A-z]` covers ASCII 65 to 122, which includes `[ \ ] ^ _` and the backtick. Use `[A-Za-z]`, or `[^\W\d_]` for any Unicode letter. ### unescaped-dot `.` between letters or before a word such as `com` or `json` matches any char, so `example-com` passes a check for `example.com`. Write `\.`. ### matches-empty The pattern accepts the empty string. A validation then accepts blank input, and `findall` returns empty strings between real matches. Use `+` instead of `*` where at least one char is required. ## INFO ### not-anchored `re.search(r"\d+", "abc123xyz")` succeeds. For validating a whole value, use `re.fullmatch` or anchor with `\A ... \Z`. ### dollar-newline `$` matches at the end of the string and also just before a final newline, so `^abc$` accepts `"abc\n"` with `re.search` or `re.match`. Use `\Z` or `re.fullmatch` for strict checks. ### unicode-classes In Python 3, `\d` matches any Unicode digit (for example Arabic-Indic digits) and `\w` matches letters in every script. Good for names and free text; for IDs, card numbers, and codes use `re.ASCII` or `[0-9]`. ### greedy-dotstar and many-dotstar `.*` takes everything and then gives back one char at a time. Three or more `.*` in one pattern make failures slow on long lines. Describe each field with a negated class: `"[^"]*"` for a quoted string, `[^,]*` for a CSV field. ## Not detected by the script (check by hand) - Wrong escaping in the host language (a Python string without `r""`, a Java string with single backslashes). - Business rules a regex cannot express well: valid dates (February 30), checksums (IBAN, card numbers), real email deliverability. - Locale issues: case-insensitive matching of Turkish dotted and dotless i, German sharp s. - Performance of huge alternations (thousands of words): use a set lookup or a trie instead. FILE:references/flavor-differences.md # Regex flavor differences that break fixes The checker runs Python `re`. Before handing a pattern to another engine, check this table. | Feature | Python `re` | JavaScript | PCRE / PHP | Go (RE2) | .NET | Java | POSIX grep -E | |---|---|---|---|---|---|---|---| | Lookahead `(?=...)` | yes | yes | yes | no | yes | yes | no | | Lookbehind `(?<=...)` | fixed width only | yes (ES2018) | fixed or bounded | no | any width | bounded | no | | Named group | `(?P<name>...)` only | `(?<name>...)` | both | `(?P<name>...)` (also `(?<name>...)` in Go 1.22+) | `(?<name>...)` | `(?<name>...)` | no | | Atomic group `(?>...)` | 3.11+ | no | yes | no | yes | yes | no | | Possessive `a++` | 3.11+ | no | yes | no | no | yes | no | | Backreference `\1` | yes | yes | yes | no | yes | yes | yes (basic) | | `\d` matches non-ASCII digits | yes (unless re.ASCII) | no | no (unless UCP) | no | yes | no (unless UNICODE_CHARACTER_CLASS) | locale | | `$` before final newline | yes | no (without m flag) | yes | no | yes | yes | n/a (line based) | | Inline flags `(?i)` | yes, at start or scoped `(?i:...)` | flags after the literal (`/i`); scoped `(?i:...)` only in the newest engines | yes | yes | yes | yes | no | | Catastrophic backtracking | possible | possible | possible (has limits) | impossible (linear time) | possible (timeout option) | possible | engine dependent | | Full match helper | `re.fullmatch` | anchor with `^...$` | anchor | anchor | anchor | `String.matches()` | `-x` flag | ## Translation notes - JavaScript: write the literal as `/pattern/flags`; escape `/` inside it. Use the `u` flag for Unicode and `v` for set operations in newer engines. `\Z` does not exist: use `$` without the `m` flag. - Go RE2: no lookarounds or backreferences, but every pattern runs in linear time, so ReDoS is not a concern there. Rewrite lookarounds as a second check in code. - Java and C#: every backslash doubles in a normal string literal (`"\\d+"`); C# verbatim strings `@"\d+"` and Java text blocks help. - .NET: set `RegexOptions.NonBacktracking` (.NET 7+) or a match timeout for untrusted input. - SQL: PostgreSQL `~` uses Tcl-style advanced regexes (lookahead, lookbehind since 9.6, no named groups); MySQL 8 uses ICU. Test in the database itself. - Shell: `grep -E` has no `\d` on many systems; use `[0-9]`. `grep -P` gives PCRE where available. ## Safe defaults across flavors - `[0-9]` instead of `\d`, `[A-Za-z]` instead of `\w` when the format is ASCII by definition. - Anchors `^` and `$` without multiline mode, plus a separate length check (for example at most 254 chars for an email). - No nested quantifiers on untrusted input, whatever the engine. FILE:templates/regex-review.md # Regex review: purpose **Pattern (original):** `original_pattern` **Engine:** language_and_engine | **Used as:** validate whole value / find first / find all / replace **Checker:** HIGH high / WARN warn / INFO info before; expectations pass_before/total pass ## What it does, in plain words 1. (one line per token or group, top to bottom) ## Test results | # | Input | Expected | Before | After | Notes | |---|---|---|---|---|---| | 1 | | match / no match | | | | ## Risks found | Level | Rule | What it means here | Fix | |---|---|---|---| | HIGH | | | | Probe (if run): input `probe_input`, probe_result ## Corrected pattern ```language (the corrected pattern, escaped for the user's language, with a VERBOSE or commented version if it is longer than about 40 chars) ``` Why each change: (one line per change) ## Keep these tests ``` + (must match) - (must not match) ``` ## Notes - Limits of a regex for this job and the extra check to do in code (length limit, parsing, verification step). - Flavor caveats if the pattern will run outside Python. FILE:examples/example-email-validation-review.md # Example: review of a signup form email check that hangs ## User request "Our signup API sometimes times out and the profiler points at this line. Can you explain the regex and fix it? It runs in Python 3.12 on the email field of a public form." ```python EMAIL = re.compile(r"^([a-zA-Z0-9]+\.?)+@[A-z]+.com$") ``` ## Cases used (cases.txt) The user's 5 cases plus 3 edge cases added during the review (a dash instead of the dot, an underscore in the domain, and 30 letters with no `@`, the shape a bot sent): ``` + ann@example.com + first.last@mail.example.co - bob@@example.com - no-at-sign.example.com - trailing-dot@example. - ann@example-com - ann@exa_mple.com - aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa! ``` ## Checker run (original) ``` $ python3 scripts/regex_check.py '^([a-zA-Z0-9]+\.?)+@[A-z]+.com$' --cases cases.txt --no-explain --probe Samples (search): PASS + 'ann@example.com' -> [0:15] 'ann@example.com' 1='ann' FAIL + 'first.last@mail.example.co' -> expected a match PASS - 'bob@@example.com' PASS - 'no-at-sign.example.com' PASS - 'trailing-dot@example.' FAIL - 'ann@example-com' -> [0:15] 'ann@example-com' 1='ann' (expected no match) FAIL - 'ann@exa_mple.com' -> [0:16] 'ann@exa_mple.com' 1='ann' (expected no match) TIMEOUT - 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!' -> still running after 2 s (catastrophic backtracking) Risks and suggestions: HIGH nested-quantifier a quantifier sits inside another quantifier and nothing else in the outer repeat is required ... probe 'a' * n + '!': n=12: 0.0002 s, n=16: 0.0031 s, n=20: 0.0593 s, n=24: 0.8419 s, n=28: timeout -> TIMEOUT at length 29: exponential backtracking confirmed WARN range-A-z [A-z] also matches [ \ ] ^ _ and the backtick; use [A-Za-z] WARN unescaped-dot '.' before 'com' matches any char (so a '-' or 'x' passes there too); write \. for a literal dot INFO dollar-newline $ also matches before a trailing newline ... Summary: HIGH 1 WARN 2 INFO 1 | expectations 4/8 pass | 1 sample(s) timed out ``` ## Review (filled template, shortened) **Checker:** HIGH 1 / WARN 2 / INFO 1 before; expectations 4/8 pass (one timeout). After: HIGH 0 / WARN 0, expectations 8/8 pass. What it does: start of string; one or more runs of letters or digits, each optionally followed by a dot; an `@`; one or more chars from A to z (including `[ \ ] ^ _` and the backtick); any single char; the text `com`; end of string or before a final newline. | # | Input | Expected | Before | After | Notes | |---|---|---|---|---|---| | 1 | ann@example.com | match | match | match | | | 2 | first.last@mail.example.co | match | no match | match | only `.com` and one domain label were allowed | | 3 | bob@@example.com | no match | no match | no match | | | 4 | no-at-sign.example.com | no match | no match | no match | | | 5 | trailing-dot@example. | no match | no match | no match | | | 6 | ann@example-com | no match | match | no match | added: the unescaped dot accepted `-` | | 7 | ann@exa_mple.com | no match | match | no match | added: `[A-z]` accepted `_` | | 8 | 30 x "a" then "!" | no match, fast | timeout after 2 s | no match, instant | added: the production hang | Risks: the `([a-zA-Z0-9]+\.?)+` group lets a run of letters be split in 2^n ways when the `@` is missing, so a 29-char junk input from a bot blocked a worker for more than 2 seconds; this is a ReDoS bug on a public endpoint. Corrected pattern: ```python EMAIL = re.compile(r""" [A-Za-z0-9._%+-]+ # local part: one run, dots allowed but not repeated in a nested group @ (?:[A-Za-z0-9-]+\.)+ # one or more domain labels, each ending in a literal dot [A-Za-z]{2,} # top-level domain """, re.VERBOSE | re.ASCII) def looks_like_email(value: str) -> bool: return len(value) <= 254 and EMAIL.fullmatch(value) is not None ``` Why each change: one flat repeat for the local part removes the nested quantifier; `\.` makes the dot literal; `[A-Za-z]` replaces `[A-z]`; any top-level domain is accepted instead of only `com`; `fullmatch` replaces `^...$`; a 254-char limit is a second defense; `re.ASCII` keeps `[...]` classes predictable. Recheck: `python3 scripts/regex_check.py '[A-Za-z0-9._%+-]+@(?:[A-Za-z0-9-]+\.)+[A-Za-z]{2,}' --cases cases.txt --mode fullmatch --probe` reports no HIGH or WARN risks (the nested group is reported as INFO nested-separated, a safe shape) and 8/8 expectations pass. Keep these tests: all 8 rows above, plus a 300-char input to check the length limit. Note that a regex only checks the shape; confirm the address by sending a verification email. FILE:scripts/regex_check.py #!/usr/bin/env python3 """Explain a Python regular expression, test it on sample inputs, and warn about risky patterns. Usage: python3 regex_check.py PATTERN [options] python3 regex_check.py PATTERN -t "text one" -t "text two" python3 regex_check.py PATTERN --cases cases.txt --mode fullmatch python3 regex_check.py PATTERN --cases - < cases.txt (read cases from stdin) Options: -t, --text TEXT a sample input (repeatable) --cases FILE one sample per line; "+ text" must match, "- text" must not match, "# ..." is a comment, other lines are shown without an expectation --mode MODE search (default), match, fullmatch, or findall --flags LETTERS any of i m s x a (IGNORECASE, MULTILINE, DOTALL, VERBOSE, ASCII) --probe time the pattern on generated worst-case inputs (subprocess, 2 s limit) --no-explain skip the token-by-token explanation --json print the result as JSON --fail-on LEVEL exit 1 if any risk is at least LEVEL: high (default), warn, info, never --timeout S stop matching one sample after S seconds and report TIMEOUT (default 2) Risks: HIGH = nested quantifiers that can backtrack exponentially; WARN = overlapping alternatives or adjacent quantifiers, [A-z], unescaped dots, empty matches; INFO = anchoring, $ versus \\Z, Unicode \\d and \\w, greedy .*, flags that look missing. Exit codes: 0 ok, 1 failed expectations or a risk at or above --fail-on, 2 usage or pattern error. Python 3 standard library only (uses the re module's own parser). """ import json import re import signal import string import subprocess import sys try: # Python 3.11+ import re._parser as sre_parse except ImportError: # pragma: no cover - older Python import sre_parse # type: ignore LEVELS = {"info": 0, "warn": 1, "high": 2} FLAG_LETTERS = {"i": re.IGNORECASE, "m": re.MULTILINE, "s": re.DOTALL, "x": re.VERBOSE, "a": re.ASCII} FLAG_NAMES = [(re.IGNORECASE, "IGNORECASE"), (re.MULTILINE, "MULTILINE"), (re.DOTALL, "DOTALL"), (re.VERBOSE, "VERBOSE"), (re.ASCII, "ASCII")] ALPHABET = string.ascii_letters + string.digits + string.punctuation + " \t\n" + "\u00e9\u0663" CATEGORY_RE = {"CATEGORY_DIGIT": r"\d", "CATEGORY_NOT_DIGIT": r"\D", "CATEGORY_SPACE": r"\s", "CATEGORY_NOT_SPACE": r"\S", "CATEGORY_WORD": r"\w", "CATEGORY_NOT_WORD": r"\W"} CATEGORY_TEXT = {"CATEGORY_DIGIT": "a digit", "CATEGORY_NOT_DIGIT": "a non-digit", "CATEGORY_SPACE": "whitespace", "CATEGORY_NOT_SPACE": "a non-whitespace char", "CATEGORY_WORD": "a word char (letter, digit, _)", "CATEGORY_NOT_WORD": "a non-word char"} AT_TEXT = {"AT_BEGINNING": "^ start of string (start of each line with MULTILINE)", "AT_BEGINNING_STRING": "\\A start of string only", "AT_END": "$ end of string or before a final newline (end of each line with MULTILINE)", "AT_END_STRING": "\\Z end of string only", "AT_BOUNDARY": "\\b word boundary", "AT_NON_BOUNDARY": "\\B not a word boundary"} MAXREPEAT = sre_parse.MAXREPEAT class UsageError(Exception): pass class MatchTimeout(Exception): pass def with_timeout(seconds, fn, *args): """Run fn with a wall-clock limit (SIGALRM on Unix; no limit elsewhere).""" if not hasattr(signal, "setitimer") or seconds <= 0: return fn(*args) def handler(_sig, _frame): raise MatchTimeout() old = signal.signal(signal.SIGALRM, handler) signal.setitimer(signal.ITIMER_REAL, seconds) try: return fn(*args) finally: signal.setitimer(signal.ITIMER_REAL, 0) signal.signal(signal.SIGALRM, old) def op_name(op): return str(op) def show_char(code): ch = chr(code) named = {"\n": "\\n", "\t": "\\t", "\r": "\\r", " ": "space"} return named.get(ch, repr(ch) if ch.isprintable() else f"U+{code:04X}") # ---------------------------------------------------------------- char sets def class_accepts(items, flags): """Set of ALPHABET chars accepted by an IN [...] item list.""" negate = bool(items) and op_name(items[0][0]) == "NEGATE" acc = set() for op, av in items: name = op_name(op) if name == "LITERAL": acc.add(chr(av)) elif name == "RANGE": acc |= {c for c in ALPHABET if av[0] <= ord(c) <= av[1]} elif name == "CATEGORY": rx = re.compile(CATEGORY_RE.get(op_name(av), "."), flags & re.ASCII) acc |= {c for c in ALPHABET if rx.match(c)} if flags & re.IGNORECASE: acc |= {c.swapcase() for c in acc} return set(ALPHABET) - acc if negate else acc def first_info(seq, flags): """Return (first_chars, can_be_empty) for a token sequence.""" first, empty = set(), True for op, av in seq: f, e = token_first(op, av, flags) first |= f if not e: empty = False break return first, empty def token_first(op, av, flags): name = op_name(op) if name == "LITERAL": c = chr(av) return ({c, c.swapcase()} if flags & re.IGNORECASE else {c}), False if name == "NOT_LITERAL": return set(ALPHABET) - {chr(av)}, False if name == "IN": return class_accepts(av, flags), False if name == "ANY": return (set(ALPHABET) if flags & re.DOTALL else set(ALPHABET) - {"\n"}), False if name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): lo, _hi, sub = av f, e = first_info(sub, flags) return f, e or lo == 0 if name == "SUBPATTERN": return first_info(av[3], flags_for(av, flags)) if name == "ATOMIC_GROUP": return first_info(av, flags) if name == "BRANCH": first, empty = set(), False for b in av[1]: f, e = first_info(b, flags) first |= f empty = empty or e return first, empty if name == "GROUPREF_EXISTS": _g, yes, no = av f1, e1 = first_info(yes, flags) f2, e2 = first_info(no or [], flags) return f1 | f2, e1 or e2 if name == "GROUPREF": return set(ALPHABET), True return set(), True # AT, ASSERT, ASSERT_NOT: zero width def flags_for(av, flags): add, delete = av[1], av[2] return (flags | add) & ~delete # ---------------------------------------------------------------- explanation def quant_text(lo, hi, name): mode = {"MAX_REPEAT": "greedy", "MIN_REPEAT": "lazy", "POSSESSIVE_REPEAT": "possessive"}[name] if hi == MAXREPEAT: base = {0: "zero or more", 1: "one or more"}.get(lo, f"{lo} or more") elif lo == hi: return f"exactly {lo}" elif (lo, hi) == (0, 1): base = "optional (zero or one)" else: base = f"{lo} to {hi}" return f"{base} ({mode})" def class_text(items): negate = bool(items) and op_name(items[0][0]) == "NEGATE" parts = [] for op, av in items: name = op_name(op) if name == "LITERAL": parts.append(show_char(av)) elif name == "RANGE": parts.append(f"{show_char(av[0])}-{show_char(av[1])}") elif name == "CATEGORY": parts.append(CATEGORY_TEXT.get(op_name(av), op_name(av))) return ("any char except " if negate else "one of: ") + ", ".join(parts) def explain(seq, group_names, flags, depth=0, out=None): out = [] if out is None else out pad = " " * depth i = 0 while i < len(seq): op, av = seq[i] name = op_name(op) if name == "LITERAL": # merge runs of literals j = i while j < len(seq) and op_name(seq[j][0]) == "LITERAL": j += 1 text = "".join(chr(a) for _, a in seq[i:j]) out.append(f"{pad}the text {text!r}" if j - i > 1 else f"{pad}the char {show_char(av)}") i = j continue if name == "NOT_LITERAL": out.append(f"{pad}any char except {show_char(av)}") elif name == "IN": out.append(f"{pad}{class_text(av)}") elif name == "ANY": out.append(f"{pad}any char" + ("" if flags & re.DOTALL else " except newline")) elif name == "AT": out.append(f"{pad}{AT_TEXT.get(op_name(av), op_name(av))}") elif name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): lo, hi, sub = av out.append(f"{pad}{quant_text(lo, hi, name)} of:") explain(sub, group_names, flags, depth + 1, out) elif name == "SUBPATTERN": g = av[0] if g is None: out.append(f"{pad}non-capturing group:") else: label = f' "{group_names[g]}"' if g in group_names else "" out.append(f"{pad}capturing group {g}{label}:") explain(av[3], group_names, flags_for(av, flags), depth + 1, out) elif name == "ATOMIC_GROUP": out.append(f"{pad}atomic group (no backtracking into it):") explain(av, group_names, flags, depth + 1, out) elif name == "BRANCH": out.append(f"{pad}either:") for k, b in enumerate(av[1]): out.append(f"{pad} option {k + 1}:") explain(b, group_names, flags, depth + 2, out) elif name in ("ASSERT", "ASSERT_NOT"): direction, sub = av kind = ("lookbehind" if direction < 0 else "lookahead") neg = "negative " if name == "ASSERT_NOT" else "" out.append(f"{pad}{neg}{kind} (checks without consuming):") explain(sub, group_names, flags, depth + 1, out) elif name == "GROUPREF": out.append(f"{pad}the same text as group {av} matched") elif name == "GROUPREF_EXISTS": out.append(f"{pad}if group {av[0]} matched:") explain(av[1], group_names, flags, depth + 1, out) if av[2]: out.append(f"{pad}otherwise:") explain(av[2], group_names, flags, depth + 1, out) else: out.append(f"{pad}{name}") i += 1 return out # ---------------------------------------------------------------- risk analysis def walk_repeats(seq, flags, path=""): """Yield (op, av, flags) for every repeat node, outside lookarounds too.""" for op, av in seq: name = op_name(op) if name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): yield op, av, flags yield from walk_repeats(av[2], flags) elif name == "SUBPATTERN": yield from walk_repeats(av[3], flags_for(av, flags)) elif name == "ATOMIC_GROUP": yield from walk_repeats(av, flags) elif name == "BRANCH": for b in av[1]: yield from walk_repeats(b, flags) elif name in ("ASSERT", "ASSERT_NOT"): yield from walk_repeats(av[1], flags) def flatten(seq): """Expand groups so the body of a repeat can be read as one token list.""" out = [] for op, av in seq: name = op_name(op) if name == "SUBPATTERN" and av[1] == 0 and av[2] == 0: out.extend(flatten(av[3])) else: out.append((op, av)) return out def token_width(tok, parsed): return sre_parse.SubPattern(parsed.state, [tok]).getwidth() def all_chars(seq, flags): """Every ALPHABET char a token sequence can consume anywhere (not only first).""" acc = set() for op, av in seq: name = op_name(op) if name in ("LITERAL", "NOT_LITERAL", "IN", "ANY"): acc |= token_first(op, av, flags)[0] elif name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): acc |= all_chars(av[2], flags) elif name == "SUBPATTERN": acc |= all_chars(av[3], flags_for(av, flags)) elif name == "ATOMIC_GROUP": acc |= all_chars(av, flags) elif name == "BRANCH": for b in av[1]: acc |= all_chars(b, flags) elif name == "GROUPREF_EXISTS": acc |= all_chars(av[1], flags) | all_chars(av[2] or [], flags) elif name == "GROUPREF": acc |= set(ALPHABET) return acc def has_unbounded(tok): op, av = tok name = op_name(op) if name in ("MAX_REPEAT", "MIN_REPEAT"): return av[1] == MAXREPEAT or has_unbounded_seq(av[2]) if name == "SUBPATTERN": return has_unbounded_seq(av[3]) if name == "BRANCH": return any(has_unbounded_seq(b) for b in av[1]) return False def has_unbounded_seq(seq): return any(has_unbounded(t) for t in seq) def sample_char(chars): for pool in (string.ascii_lowercase, string.digits, string.ascii_uppercase, ALPHABET): for c in pool: if c in chars: return c return None def analyze(parsed, flags, pattern, modes_seen): risks = [] def add(level, rule, msg, probe=None): risks.append({"level": level, "rule": rule, "message": msg, **({"probe": probe} if probe else {})}) # nested quantifiers: a repeat whose body can match the same text in more than one way for op, av, fl in walk_repeats(list(parsed), flags): if op_name(op) == "POSSESSIVE_REPEAT" or av[1] <= 1: continue body = flatten(av[2]) widths = [token_width(t, parsed) for t in body] if sum(w[0] for w in widths) == sum(w[1] for w in widths): continue # fixed-width body such as (\d{4}-)+ cannot be split two ways variable = [t for t, w in zip(body, widths) if w[0] != w[1]] fixed = [t for t, w in zip(body, widths) if w[0] == w[1] and w[0] > 0] var_chars = set().union(*(all_chars([t], fl) for t in variable)) has_inner_loop = any(has_unbounded(t) for t in variable) overlap = set().union(*(all_chars([t], fl) for t in fixed)) & var_chars if fixed else set() sample = sample_char(overlap) or sample_char(var_chars) if not fixed and has_inner_loop: add("high", "nested-quantifier", "a quantifier sits inside another quantifier and nothing else in the outer repeat is " "required (the classic (a+)+ or (\\w+\\s?)+ shape), so one string can be split in " "exponentially many ways; rewrite as a single repeat, make the separator required, or use an " "atomic group or possessive quantifier (Python 3.11+)", probe=sample) elif any(not (all_chars([t], fl) & var_chars) for t in fixed): if has_inner_loop: add("info", "nested-separated", "nested quantifier with a required separator that the inner repeat cannot match " "(like ([\\w-]+\\.)+); this shape is safe") elif all(all_chars([t], fl) <= var_chars for t in fixed): add("high", "ambiguous-repeat", "the body of a repeat has variable length and every required part can also be eaten by " "the variable part (for example (x+x+)+, (aa|a)+, or (\\w+|\\d+)+), so the iterations " "can be split in exponentially many ways; merge the parts into one class or make each " "iteration start with a char the rest cannot match", probe=sample) else: add("warn", "nested-overlap", "a repeat with a variable-length body whose separator is also accepted by the inner part " "(for example ([\\w.]+\\.)+): long non-matching input can slow down sharply; make the inner " "class exclude the separator", probe=sample) # adjacent overlapping unbounded repeats in any sequence seen_pairs = set() def single_char_repeat(op, av): return (op_name(op) in ("MAX_REPEAT", "MIN_REPEAT") and av[1] == MAXREPEAT and len(av[2]) == 1 and op_name(av[2][0][0]) in ("LITERAL", "NOT_LITERAL", "IN", "ANY")) def scan_adjacent(seq, fl): toks = [t for t in flatten(seq) if op_name(t[0]) not in ("AT", "ASSERT", "ASSERT_NOT")] for (a_op, a_av), (b_op, b_av) in zip(toks, toks[1:]): if single_char_repeat(a_op, a_av) and single_char_repeat(b_op, b_av): if (id(a_av), id(b_av)) in seen_pairs: continue seen_pairs.add((id(a_av), id(b_av))) fa, _ = first_info(a_av[2], fl) fb, _ = first_info(b_av[2], fl) if fa & fb: add("warn", "adjacent-overlap", "two unbounded quantifiers in a row accept the same chars (for example \\d+\\d+ or " ".*.*), so the split between them is ambiguous and failures backtrack polynomially; " "keep one of them") for op, av in seq: name = op_name(op) if name == "SUBPATTERN": scan_adjacent(av[3], flags_for(av, fl)) elif name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): scan_adjacent(av[2], fl) elif name == "BRANCH": for b in av[1]: scan_adjacent(b, fl) scan_adjacent(list(parsed), flags) # text-level checks if re.search(r"\[[^\]]*A-z", pattern): add("warn", "range-A-z", "[A-z] also matches [ \\ ] ^ _ and the backtick; use [A-Za-z]") seq = list(parsed) flat = flatten(seq) for k, (op, _av) in enumerate(flat): if op_name(op) != "ANY": continue before = flat[k - 1] if k else None after = [chr(a) for o, a in flat[k + 1:k + 4] if op_name(o) == "LITERAL"] lit_before = before is not None and op_name(before[0]) == "LITERAL" and chr(before[1]).isalnum() word_after = len(after) >= 2 and all(c.isalpha() for c in after[:2]) if (lit_before and after and after[0].isalnum()) or word_after: add("warn", "unescaped-dot", f"'.' before {''.join(after) or 'the next token'!r} matches any char (so a '-' or 'x' passes there too); " "write \\. for a literal dot") break try: if re.compile(pattern, flags).fullmatch(""): add("warn", "matches-empty", "the pattern matches the empty string, so findall/finditer return empty matches and " "validation accepts blank input; require at least one char where it matters") except re.error: pass count_dotstar = len(re.findall(r"(?<!\\)\.\*", pattern)) if count_dotstar >= 3: add("info", "many-dotstar", f"{count_dotstar} '.*' in one pattern: failures are slow on long lines; " "use negated classes like [^,]* to describe each field") elif count_dotstar and not re.search(r"\.\*\?", pattern): add("info", "greedy-dotstar", "greedy .* grabs as much as possible and then backs off; use .*? or a " "negated class such as [^\"]* when you mean 'up to the next ...'") first_ops = [op_name(op) for op, _ in seq[:1]] last = seq[-1] if seq else None anchored_start = first_ops == ["AT"] anchored_end = last is not None and op_name(last[0]) == "AT" and op_name(last[1]) in ("AT_END", "AT_END_STRING") if "fullmatch" not in modes_seen and not (anchored_start and anchored_end) and modes_seen != {"findall"}: add("info", "not-anchored", "for validation, anchor both ends (\\A ... \\Z) or use re.fullmatch; " "otherwise 'abc123xyz' passes a search for \\d+") if anchored_end and op_name(last[1]) == "AT_END" and not flags & re.MULTILINE and "fullmatch" not in modes_seen: add("info", "dollar-newline", "$ also matches before a trailing newline ('abc\\n' passes ^abc$); " "use \\Z or fullmatch for strict validation") if re.search(r"(?<!\\)\\[dw]", pattern) and not flags & re.ASCII: add("info", "unicode-classes", "\\d and \\w match Unicode digits and letters too (for example " "Arabic-Indic digits); add re.ASCII or use [0-9] / [A-Za-z0-9_] for strict input formats") return risks # ---------------------------------------------------------------- probe PROBE_CODE = r""" import re, sys, time pat, flags, s = sys.argv[1], int(sys.argv[2]), sys.argv[3] rx = re.compile(pat, flags) t = time.perf_counter(); rx.search(s); print(f"{time.perf_counter() - t:.6f}") """ def run_probe(pattern, flags, ch, compiled): tail = next((t for t in "!#\x00~ \n" if not compiled.search(ch * 3 + t)), "!") results, verdict = [], "no slowdown seen" for n in (12, 16, 20, 24, 28): s = ch * n + tail try: r = subprocess.run([sys.executable, "-c", PROBE_CODE, pattern, str(flags), s], capture_output=True, text=True, timeout=2) secs = float(r.stdout.strip() or "nan") except subprocess.TimeoutExpired: results.append({"n": n, "seconds": None}) verdict = f"TIMEOUT at length {n + 1}: exponential backtracking confirmed" break results.append({"n": n, "seconds": round(secs, 4)}) prev = results[-2]["seconds"] if len(results) >= 2 else None if prev and secs > 0.01 and secs / prev > 3: verdict = (f"time grows about {secs / prev:.0f}x per +4 chars: exponential backtracking confirmed") elif secs > 0.5: verdict = f"slow ({secs:.2f} s at length {n + 1}) but not clearly exponential" return {"input": f"{ch!r} * n + {tail!r}", "timings": results, "verdict": verdict} # ---------------------------------------------------------------- matching def read_cases(path): text = sys.stdin.read() if path == "-" else open(path, encoding="utf-8").read() cases = [] for raw in text.splitlines(): if not raw.strip() or raw.lstrip().startswith("#"): continue if raw.startswith("+ ") or raw.startswith("- "): cases.append({"text": raw[2:], "expect": raw[0] == "+"}) else: cases.append({"text": raw, "expect": None}) return cases def match_case(rx, text, mode): fn = {"search": rx.search, "match": rx.match, "fullmatch": rx.fullmatch}.get(mode) names = {v: k for k, v in rx.groupindex.items()} def groups(m): out = [] for g in range(1, rx.groups + 1): out.append({"group": g, "name": names.get(g), "value": m.group(g), "span": list(m.span(g)) if m.group(g) is not None else None}) return out if mode == "findall": ms = list(rx.finditer(text)) return {"matched": bool(ms), "matches": [{"text": m.group(0), "span": list(m.span()), "groups": groups(m)} for m in ms]} m = fn(text) if not m: return {"matched": False, "matches": []} return {"matched": True, "matches": [{"text": m.group(0), "span": list(m.span()), "groups": groups(m)}]} def describe_flags(flags): names = [n for f, n in FLAG_NAMES if flags & f] return ", ".join(names) if names else "none" # ---------------------------------------------------------------- main def parse_args(argv): opts = {"texts": [], "cases": None, "mode": "search", "flags": 0, "probe": False, "explain": True, "json": False, "fail_on": "high", "pattern": None, "timeout": 2.0} i = 0 while i < len(argv): a = argv[i] def val(): if i + 1 >= len(argv): raise UsageError(f"{a} needs a value") return argv[i + 1] if a in ("-t", "--text"): opts["texts"].append(val()); i += 2 elif a == "--cases": opts["cases"] = val(); i += 2 elif a == "--mode": opts["mode"] = val(); i += 2 if opts["mode"] not in ("search", "match", "fullmatch", "findall"): raise UsageError("--mode must be search, match, fullmatch, or findall") elif a == "--flags": letters = val(); i += 2 bad = [c for c in letters.lower() if c not in FLAG_LETTERS] if bad: raise UsageError(f"unknown flag letter(s) {''.join(bad)!r}; use any of i m s x a") for c in letters.lower(): opts["flags"] |= FLAG_LETTERS[c] elif a == "--fail-on": opts["fail_on"] = val().lower(); i += 2 if opts["fail_on"] not in ("high", "warn", "info", "never"): raise UsageError("--fail-on must be high, warn, info, or never") elif a == "--timeout": try: opts["timeout"] = float(val()) except ValueError: raise UsageError("--timeout needs a number of seconds, for example --timeout 2") i += 2 elif a == "--probe": opts["probe"] = True; i += 1 elif a == "--no-explain": opts["explain"] = False; i += 1 elif a == "--json": opts["json"] = True; i += 1 elif a in ("-h", "--help"): print(__doc__); sys.exit(0) elif a.startswith("--") and opts["pattern"] is not None: raise UsageError(f"unknown option {a}") elif opts["pattern"] is None: opts["pattern"] = a; i += 1 else: raise UsageError(f"unexpected argument {a!r} (quote the pattern; use -t for sample inputs)") if opts["pattern"] is None: raise UsageError("give a PATTERN (quote it in single quotes in the shell)") return opts def main(argv): try: opts = parse_args(argv) cases = [{"text": t, "expect": None} for t in opts["texts"]] if opts["cases"]: try: cases += read_cases(opts["cases"]) except OSError as e: raise UsageError(f"cannot read cases file: {e.strerror}: {opts['cases']}") except UsageError as e: print(f"error: {e}", file=sys.stderr) return 2 pattern, flags, mode = opts["pattern"], opts["flags"], opts["mode"] try: rx = re.compile(pattern, flags) parsed = sre_parse.parse(pattern, flags) except re.error as e: pos = f" at position {e.pos}" if e.pos is not None else "" print(f"error: pattern does not compile: {e.msg}{pos}", file=sys.stderr) if e.pos is not None: print(f" {pattern}\n {' ' * e.pos}^", file=sys.stderr) return 2 flags = int(rx.flags) & ~int(re.UNICODE) names = {v: k for k, v in rx.groupindex.items()} risks = analyze(parsed, flags, pattern, {mode}) if opts["probe"]: for r in risks: if r.get("probe"): r["probe_result"] = run_probe(pattern, flags, r["probe"], rx) results = [] for c in cases: try: res = with_timeout(opts["timeout"], match_case, rx, c["text"], mode) status = None if c["expect"] is None else ("PASS" if res["matched"] == c["expect"] else "FAIL") except MatchTimeout: res, status = {"matched": None, "matches": []}, "TIMEOUT" results.append({**c, **res, "status": status}) fails = sum(1 for r in results if r["status"] in ("FAIL", "TIMEOUT")) counts = {lvl: sum(1 for r in risks if r["level"] == lvl) for lvl in ("high", "warn", "info")} report = {"pattern": pattern, "flags": describe_flags(flags), "mode": mode, "groups": rx.groups, "group_names": {str(k): v for k, v in names.items()}, "explanation": explain(list(parsed), names, flags) if opts["explain"] else [], "cases": results, "failed_expectations": fails, "risks": risks, "risk_counts": counts} threshold = LEVELS.get(opts["fail_on"], 99) risky = any(LEVELS[r["level"]] >= threshold for r in risks) if opts["json"]: print(json.dumps(report, indent=2, ensure_ascii=False)) return 1 if fails or risky else 0 print(f"Pattern: {pattern}") named = ", ".join(f'{g} "{n}"' for g, n in sorted(names.items())) print(f"Flags: {describe_flags(flags)} | mode: {mode} | groups: {rx.groups}" + (f" ({named})" if named else "")) if opts["explain"]: print("\nExplanation:") for line in report["explanation"]: print(" " + line) if results: print(f"\nSamples ({mode}):") for r in results: mark = {True: "+", False: "-", None: " "}[r["expect"]] tag = r["status"] or ("match" if r["matched"] else "no match") line = f" {tag:<8} {mark} {r['text']!r}" if r["matched"]: parts = [] for m in r["matches"][:5]: g = " ".join(f"{x['name'] or x['group']}={x['value']!r}" for x in m["groups"]) parts.append(f"[{m['span'][0]}:{m['span'][1]}] {m['text']!r}" + (f" {g}" if g else "")) more = f" (+{len(r['matches']) - 5} more)" if len(r["matches"]) > 5 else "" line += " -> " + "; ".join(parts) + more elif r["status"] == "TIMEOUT": line += f" -> still running after {opts['timeout']:g} s (catastrophic backtracking)" elif r["status"] == "FAIL": line += " -> expected a match" if r["status"] == "FAIL" and r["matched"]: line += " (expected no match)" print(line) print("\nRisks and suggestions:") if not risks: print(" none found") for r in risks: print(f" {r['level'].upper():<4} {r['rule']:<23} {r['message']}") pr = r.get("probe_result") if pr: times = ", ".join(f"n={t['n']}: {'timeout' if t['seconds'] is None else str(t['seconds']) + ' s'}" for t in pr["timings"]) print(f" probe {pr['input']}: {times} -> {pr['verdict']}") summary = f"\nSummary: HIGH {counts['high']} WARN {counts['warn']} INFO {counts['info']}" if any(r["expect"] is not None for r in results): n = sum(1 for r in results if r["expect"] is not None) summary += f" | expectations {n - fails}/{n} pass" timeouts = sum(1 for r in results if r["status"] == "TIMEOUT") if timeouts: summary += f" | {timeouts} sample(s) timed out" print(summary) return 1 if fails or risky else 0 if __name__ == "__main__": sys.exit(main(sys.argv[1:]))
【中文说明】用于把播客逐字稿、时间戳和嘉宾资料整理成可直接发布的节目说明，包括标题、SEO 摘要、重点、章节、引用、嘉宾简介、链接和社交媒体文案。使用时粘贴完整 transcript、时间戳及嘉宾信息；若有专有名词、平台限制或目标受众，也应一并说明以便校正输出。 Turns a rough podcast transcript, timestamps, and guest notes into publish-ready show notes: an episode title, an SEO summary, key takeaways, chapter markers for YouTube, Spotify, and Podcasting 2.0, timed pull quotes, a guest bio, links, and social posts. Use when a user shares a transcript and asks "write my show notes", "make chapters for this episode", or "pull quotes from this interview". Includes a tested stdlib Python chapter finder.
---
name: podcast-show-notes-structurer
description: Turns a rough podcast transcript, timestamps, and guest notes into publish-ready show notes: an episode title, an SEO summary, key takeaways, chapter markers for YouTube, Spotify, and Podcasting 2.0, timed pull quotes, a guest bio, links, and social posts. Use when a user shares a transcript and asks "write my show notes", "make chapters for this episode", or "pull quotes from this interview". Includes a tested stdlib Python chapter finder.
---
# Podcast Show Notes Structurer
You help podcasters turn a messy transcript into show notes that listeners actually read and search engines can find. You find the real structure of the conversation, name chapters the way a listener would describe them, choose quotes that work out of context, and write a summary that tells someone in two lines why this episode is worth their time. You never invent facts, links, or quotes that are not in the material.
## Files in this skill
- `scripts/chapterize.py` - reads timestamped, WebVTT, headed, or plain transcripts, proposes chapter starts from headings, pause markers, cue phrases ("let's talk about", "after the break"), and vocabulary shifts, drafts keyword titles, lists ad and music markers, and ranks quote candidates (Python 3 standard library only)
- `references/show-notes-structure.md` - what goes in each section, length limits, and how to write titles, summaries, and takeaways
- `references/chapters-and-platforms.md` - chapter rules for YouTube, Spotify, Apple Podcasts, and Podcasting 2.0, plus quote and timestamp ethics
- `templates/show-notes.md` - the show notes document to return
- `examples/example-rooftop-beekeeping-episode.md` - a worked example from a 25-minute interview
## Workflow
### 1. Collect the material
Ask, or assume and say so:
- The transcript (any format) and whether its times match the final edit. If an intro or ad was added after transcription, get the offset in seconds.
- Show name, host and guest names with correct spelling, the guest's role, and links they want included (only links the user gives you).
- Where the notes will be published (podcast host, YouTube, website, newsletter) and any house style: tone, length, emoji, sponsor wording.
- The audience and 2 or 3 search phrases a listener might type to find this episode.
### 2. Run the chapter finder
```bash
python3 scripts/chapterize.py transcript.txt
python3 scripts/chapterize.py transcript.txt --min-chapter 3 --offset 45 --duration 52:30
python3 scripts/chapterize.py episode.vtt --format youtube
python3 scripts/chapterize.py - --format json < transcript.txt
```
The script's chapter starts and quotes are candidates, not answers. Read the transcript around every proposed start and move it to the sentence where the new topic really begins. Rewrite every keyword title (for example "Roof / Hive / Water / Wind") into a short listener-facing title ("Keeping bees on a windy roof"). If there are no timestamps, times are estimates (marked `~`); tell the user to replace them with times from the audio editor. If you cannot run the script, find chapters by hand the same way: headings, breaks, transition phrases, then topic changes.
### 3. Write the notes
1. Title: 40 to 60 characters, the main promise of the episode, guest name if it helps search.
2. Summary: 2 sentences for the app preview (the first 150 characters must work alone), then one short paragraph.
3. Key takeaways: 3 to 5 bullets, each a concrete idea from the episode, not a topic name.
4. Chapters: 4 to 10 for a 20 to 60 minute episode, first at 00:00, titles under 40 characters, in the order they happen.
5. Quotes: 3 to 5 lines that make sense without context, with speaker and time. Remove filler words only; never change meaning. Mark edited quotes so the user can check them against the audio.
6. Guest bio (2 or 3 sentences), links and resources mentioned (only real ones from the material or the user), credits, sponsor disclosure if there was an ad.
7. Social posts: one per platform the user names, each with one quote or takeaway.
### 4. Deliver
Fill in `templates/show-notes.md` and follow `references/show-notes-structure.md` and `references/chapters-and-platforms.md`. Follow the style of `examples/example-rooftop-beekeeping-episode.md`. Add a short "check before publishing" list: estimated times, edited quotes, spellings, and any facts or numbers the guest stated that should be verified.
## Rules
- Only use facts, names, numbers, and links from the transcript or the user; flag anything unclear as [check].
- Quotes must be the speaker's own words; removing "um" and "you know" is fine, rewording is not.
- Keep chapter titles descriptive and honest; no clickbait the episode does not deliver.
- Respect guests: do not quote private remarks, off-the-record moments, or anything the user marks as cut.
- Match the show's voice; default to warm, clear, and specific.
FILE:references/show-notes-structure.md
# Show notes structure
Listeners skim show notes in a podcast app on a phone, search engines read them on the episode page, and the guest shares them. Each section has one job.
| Section | Job | Length |
|---|---|---|
| Episode title | Say the main promise; help search | 40 to 60 characters; most apps cut after about 60 |
| Preview summary | Make someone press play from the app list | First 150 characters must work alone; 2 sentences total |
| Full summary | Explain who the guest is and what you cover | 60 to 120 words |
| Key takeaways | Give skimmers the value | 3 to 5 bullets, one idea each, under 20 words |
| Chapters | Let listeners jump | 4 to 10 for 20 to 60 minutes; titles under 40 characters |
| Pull quotes | Quotable lines for notes and social | 3 to 5, 12 to 30 words each, with speaker and time |
| Guest bio | Credit the guest | 2 or 3 sentences, written in third person |
| Links and resources | Everything mentioned that has a link | Only links the user or guest gave |
| Credits and disclosure | Host, guest, producer, music, sponsor | One line each |
| Call to action | One next step | Subscribe, review, newsletter, or a question for listeners |
## Titles
- Lead with the idea, not the episode number: "Keeping Bees on a Hotel Roof with Daniel Okafor" beats "Ep. 48: Daniel Okafor".
- Use the words a listener would search: "rooftop beekeeping", "city honey".
- Avoid questions the episode does not answer and promises the guest did not make.
## Summaries
- Sentence 1: who and what, in plain words. Sentence 2: the most surprising or useful point.
- Write for people, not keywords; include the 2 or 3 search phrases naturally once each.
- No "In this episode we talk about..." filler; start with the substance.
## Key takeaways
Write each takeaway as a claim or instruction from the episode, not a topic:
- Weak: "Water on roofs."
- Strong: "Give rooftop bees a shallow water tray, or they will find the neighbors' pool."
## Chapter titles
- Describe what the listener gets in that part, in 2 to 6 words: "Why wind is the real enemy".
- Keep them parallel in style (all statements or all questions).
- Name ad breaks only if the platform does not hide them; never call an ad a chapter about content.
## Quotes
- Pick lines that stand alone: a clear claim, a lesson, a number, or a vivid image.
- Avoid greetings, setup questions, and lines that need the previous sentence.
- Removing fillers (um, uh, you know) is fine; mark the quote as lightly edited. Changing words is not.
## Accessibility and trust
- Link a full transcript when one exists; it helps deaf and hard-of-hearing listeners and search.
- Disclose sponsors and affiliate links clearly.
- Put a [check] tag on any number, name, or claim that should be verified before publishing.
FILE:references/chapters-and-platforms.md
# Chapters and platform rules
Rules change; when in doubt, check the platform's current help page. These are the stable basics.
## YouTube (video podcasts and audio uploads)
- Chapters come from timestamps in the description, one per line: `00:00 Intro`.
- The first timestamp must be 00:00.
- At least 3 timestamps, and each chapter at least 10 seconds long.
- Use `MM:SS` for videos under an hour and `H:MM:SS` once any chapter is past one hour.
- `chapterize.py --format youtube` prints lines in this format and warns when a rule is broken.
## Spotify
- Spotify reads chapter timestamps from the episode description in the same `MM:SS Title` style for episodes on its platform; follow the same rules as YouTube (start at 00:00, a few minutes apart).
## Apple Podcasts and most other apps
- Chapters are embedded in the audio file: ID3 CHAP frames in MP3 or a chapter track in M4A/AAC. Add them in the audio editor or a tagging tool before uploading.
- Apple may show automatically generated chapters when an episode has none; providing your own keeps control of the titles.
## Podcasting 2.0 (RSS)
- The `<podcast:chapters>` tag points to a JSON chapters file, supported by many modern apps and hosts:
```json
{
"version": "1.2.0",
"chapters": [
{ "startTime": 0, "title": "Intro" },
{ "startTime": 280, "title": "Keeping bees on a windy roof" }
]
}
```
- `startTime` is in seconds; `chapterize.py --format json` gives `start` in seconds for each chapter.
## Timestamps
- Transcript times often differ from the final file: intros, ads, and edits shift them. Use `--offset` for a fixed shift and recheck a few chapters in the audio.
- Without timestamps, `chapterize.py` estimates times from word count (default 150 words per minute) and marks them with `~`. Never publish estimated times without checking.
- Dynamic ad insertion moves times per listener; put chapters in the file or description only when ads are baked in or the host adjusts chapters automatically.
## How the script finds chapter starts
1. Headings (`# Title`, `Chapter 3: ...`, `Segment: ...`) always start a chapter.
2. Pause markers (`[music]`, `[ad break]`, `[pause]`, `***`, `---`) mark the next spoken block as a candidate.
3. Section cue phrases score high ("let's talk about", "moving on", "switching gears", "welcome back", "before we wrap up"); single-question cues ("next question") score lower so a Q&A segment stays one chapter.
4. Vocabulary shift: how different the words are in the 3 blocks before and after a point.
5. Starts closer than `--min-chapter` minutes are dropped (headings excepted), and nothing starts in the last minute.
## Quote and transcript ethics
- Keep quotes faithful; mark filler removal as an edit.
- Ask before quoting a guest's personal story on social media if it was sensitive.
- Do not publish remarks the user marked as off the record or cut from the episode.
FILE:templates/show-notes.md
# episode_title
**show_name** | Episode episode_number | duration | Guest: guest_name
preview_summary_two_sentences
full_summary_paragraph
## Key takeaways
-
-
-
## Chapters
```
00:00 chapter_1
time chapter_2
```
## Quotes
> "quote" - speaker (time)
## About guest_name
guest_bio
## Links and resources mentioned
- resource - url_or_check
## Credits
Host: host | Guest: guest_name | Produced by: producer | Music: music_credit
sponsor_disclosure
call_to_action
---
## Social posts
- **platform:** post_with_quote_or_takeaway
## Check before publishing
- [ ] Chapter times checked against the final audio (offset: offset)
- [ ] Edited quotes checked against the audio
- [ ] Names and spellings confirmed
- [ ] Facts and numbers marked [check] verified
- [ ] Links tested
FILE:examples/example-rooftop-beekeeping-episode.md
# Example: show notes for a 25-minute interview
## User request
"Here's the transcript of episode 48 of Small Plot Radio with Daniel Okafor, who keeps bees on a hotel roof. We added a 45-second intro in the edit, so the final file runs 26:02. Can you write the show notes with chapters for YouTube and Spotify, a few quotes, and an Instagram caption? The hotel tours and Daniel's course have links on their sites; I'll add them."
## Chapter finder run
```
$ python3 scripts/chapterize.py rooftop-bees.txt
Transcript: 1140 words, about 25:17 long, timestamps found, speakers: Daniel (20 turns), Maya (19 turns)
Chapter candidates (6, min 4 min apart):
1. 00:00 4.7 min [detected] Hotel Manager / Hive / Twelve / Bees
2. 04:40 4.5 min [detected] Roof / Hive / Water / Wind
3. 09:12 5.3 min [detected] Honey Tour / Spring Honey / Late Summer / Guests
4. 14:30 5.0 min [detected] City / Bees / Honey / Roof
5. 19:30 4.7 min [detected] Hive / Balcony / Bees / Priya
6. 24:10 1.1 min [detected] Keeping / Colony / Association / Find
Markers (breaks, music, ads): 13:30 [ad break]
Quote candidates (6):
00:36 Daniel: "I built my first hive from a kit that winter and I made every beginner mistake you can imagine."
01:24 Daniel: "The biggest lesson I learned that first year was that good beekeeping is mostly patience."
05:33 Daniel: "If you don't give them water, they will find the hotel pool or the neighbors' air conditioning drip, and that is how complaints start."
...
$ python3 scripts/chapterize.py rooftop-bees.txt --offset 45 --format youtube
00:00 Hotel Manager / Hive / Twelve / Bees
05:25 Roof / Hive / Water / Wind
09:57 Honey Tour / Spring Honey / Late Summer / Guests
15:15 City / Bees / Honey / Roof
20:15 Hive / Balcony / Bees / Priya
24:55 Keeping / Colony / Association / Find
# note: YouTube needs the first chapter at 00:00; the first one was moved to 00:00
```
Review of the candidates: all six starts land on real transitions ("Let's talk about the roof itself", "Switching gears to the hotel manager", "Welcome back", "Let's move on to listener questions", "Before we wrap up"). The 13:30 ad break stays inside chapter 3 because the ad is baked in. Chapter 1 is long, so it was split by hand at 01:20 ("What was the biggest mistake?"), which becomes 02:05 after the offset. Every keyword title was rewritten.
## Show notes (filled template)
# Keeping Bees on a Hotel Roof with Daniel Okafor
**Small Plot Radio** | Episode 48 | 26:02 | Guest: Daniel Okafor
Daniel Okafor keeps twelve beehives on a downtown hotel roof. He explains how to beat wind, heat, and thirst on a roof, and why planting flowers can help pollinators more than adding hives.
Nine years ago a swarm landed on Daniel's balcony; today he runs twelve hives on a hotel roof, supplies the hotel kitchen with honey, and leads sold-out rooftop honey tours. Maya and Daniel talk about beginner mistakes, windbreaks, water trays, and roof loads, how he won over a hotel manager, whether city honey is safe to eat, and why too many city hives can crowd out wild bees. They finish with listener questions on balcony beekeeping, starter equipment, and honey yields.
## Key takeaways
- Inspect hives every 7 to 10 days in spring, with a plan, and leave them alone in between.
- On a roof, wind is the main enemy: a 1.5 m slatted windbreak on the exposed sides makes the difference.
- Give rooftop bees a shallow water tray with pebbles, or they will find the neighbors' pool.
- Have the roof load checked first: a full hive can weigh 80 to 100 kg.
- Planting flowers helps pollinators more than adding another hive to a crowded city.
## Chapters
```
00:00 How a swarm made Daniel a beekeeper
02:05 The beginner mistake: opening the hive too often
05:25 Keeping bees on a windy roof
09:57 Winning over the hotel (and its kitchen)
15:15 Is city honey safe, and are there too many hives?
20:15 Listener questions: balconies, gear, and honey yields
24:55 The one thing every new beekeeper should know
```
## Quotes
> "The biggest lesson I learned that first year was that good beekeeping is mostly patience." - Daniel Okafor (02:09)
> "If you don't give them water, they will find the hotel pool or the neighbors' air conditioning drip, and that is how complaints start." - Daniel Okafor (06:18)
> "If you want to help pollinators, planting flowers matters more than adding another hive." - Daniel Okafor (16:53)
## About Daniel Okafor
Daniel Okafor is a city beekeeper who has kept bees for nine years and now runs twelve hives on the roof of a downtown hotel. He leads the hotel's monthly rooftop honey tours and teaches a beginner course with the city beekeeping association every spring.
## Links and resources mentioned
- Hotel rooftop honey tours - [link from user]
- Beginner beekeeping course, city beekeeping association - [link from user]
- Mason bee houses for balconies (mentioned at 20:15) - no link given
## Credits
Host: Maya | Guest: Daniel Okafor | Produced by: [check] | Music: [check]
This episode includes an advertisement at 14:15. [check sponsor name for disclosure]
Got a balcony or roof question for Daniel? Send it in and we'll answer it in a future episode.
---
## Social posts
- **Instagram:** "Planting flowers matters more than adding another hive." Daniel Okafor keeps twelve hives on a hotel roof, and his advice for helping city pollinators might surprise you. New episode of Small Plot Radio, link in bio.
## Check before publishing
- [ ] Chapter times checked against the final audio (offset: +45 s added to every transcript time)
- [ ] Quotes checked against the audio (none were edited)
- [ ] Spelling of "Okafor" confirmed with the guest
- [ ] Facts to verify: "zero stings on guests in nine seasons", yearly lab tests below limits, 15 to 30 kg per hive
- [ ] Links added and tested
FILE:scripts/chapterize.py
#!/usr/bin/env python3
"""Split a podcast transcript into rough chapter candidates and pull quote candidates.
Usage:
python3 chapterize.py TRANSCRIPT [options]
python3 chapterize.py - < transcript.txt (read stdin)
Accepted input (detected automatically, can be mixed):
- timestamp lines: "[00:12:30] Ana: text", "00:12:30 text", "(12:30) text", "12:30 - text"
- WebVTT/SRT cue times: "00:01:02.000 --> 00:01:05.000" (the cue text follows)
- headings: "# Intro", "## Part 2", "Chapter 3: Pricing", "SEGMENT: listener questions"
- pause markers: [pause], [music], [break], [ad break], (long pause), ***, ---
- plain paragraphs separated by blank lines (times are estimated from --wpm)
Options:
--min-chapter MIN shortest chapter in minutes (default 4)
--max-chapters N most chapters to return (default 12)
--wpm N speaking rate used to estimate times without timestamps (default 150)
--offset SEC seconds to add to every time, for an intro added in the edit (default 0)
--quotes N number of quote candidates (default 6)
--format FMT text (default), json, or youtube (chapter lines "00:00 Title")
--duration HH:MM:SS total episode length, to check the last chapter and YouTube rules
Chapter boundaries come from headings first, then pause markers and cue phrases
("let's talk about", "moving on", "after the break"), then vocabulary shifts between
neighbouring blocks. Titles are keyword guesses: always rewrite them for the listener.
Exit codes: 0 ok, 1 input too short for chapters, 2 usage or read error.
Python 3 standard library only.
"""
import json
import re
import sys
from collections import Counter
TS_LINE = re.compile(r"^\s*[\[(]?((?:\d{1,2}:)?\d{1,2}:\d{2})(?:[.,]\d+)?[\])]?\s*(?:[-\u2013\u2014|:]\s*)?(.*)$")
CUE_LINE = re.compile(r"^\s*((?:\d{1,2}:)?\d{1,2}:\d{2})[.,]\d{1,3}\s*-->\s*((?:\d{1,2}:)?\d{1,2}:\d{2})?")
LAST_CUE_END = [0]
HEADING = re.compile(r"^\s*(?:#{1,6}\s+(.+)|(?:chapter|part|segment|section|topic)\s*\d*\s*[:.\-\u2013]\s*(.+))\s*$", re.I)
PAUSE = re.compile(r"^\s*(?:\[(?:pause|long pause|music|break|ad break|ad|sponsor|interlude|silence)[^\]]*\]|"
r"\((?:pause|long pause|music|break|ad break|silence)[^)]*\)|\*{3,}|-{3,}|={3,})\s*$", re.I)
SPEAKER = re.compile(r"^([A-Z][\w.'-]*(?: [A-Z][\w.'-]*){0,2}):\s+(.*)$")
# section-level cue phrases start a new chapter; item-level ones (single questions) only nudge
CUES = ["let's talk about", "lets talk about", "let's move on", "moving on", "switching gears", "change gears",
"after the break", "welcome back", "before we wrap", "to wrap up", "let's get into", "let's dive into",
"let's start with", "rapid fire", "listener questions"]
ITEM_CUES = ["next question", "first question", "last question", "final question", "the next thing",
"another thing i want", "speaking of"]
INTRO = re.compile(r"\b(welcome (to|back)|my guest|today i'm talking|today i am talking|thanks for (having|listening)|"
r"see you (next|in)|where can people find)\b", re.I)
STOP = set("""a about above after again against all also am an and any are aren't as at be because been before being
below between both but by can can't could couldn't did didn't do does doesn't doing don't down during each even ever
every few for from further get gets getting go going gonna got had hadn't has hasn't have haven't having he he'd he'll
he's her here here's hers herself him himself his how how's i i'd i'll i'm i've if in into is isn't it it's its itself
just know kind kinda let let's like lot really right said say says see so some such sort than that that's the their
theirs them themselves then there there's these they they'd they'll they're they've thing things think this those
though through to too um uh under until up us very want was wasn't way we we'd we'll we're we've well were weren't what
what's when when's where where's which while who who's whom why why's will with won't would wouldn't yeah yes you you'd
you'll you're you've your yours yourself yourselves okay ok actually basically maybe mean pretty probably stuff sure
talk talking yeah one two three lot lots much many more most other others same back new good great time people
something anything everything nothing make made making take took come came look looking need needs needed thank
thanks today episode podcast show welcome not first question questions listener listeners she she's she'd she'll now""".split())
QUOTE_WORDS = re.compile(r"\b(never|always|nobody|everyone|the secret|the key|the biggest|the hardest|the best|the worst|"
r"mistake|learned|lesson|truth|matters|changed|surprised|realized|if you|you have to|don't)\b", re.I)
FILLER = re.compile(r"\b(um+|uh+|you know|i mean|sort of|kind of|like,)\s*", re.I)
class UsageError(Exception):
pass
def to_sec(ts):
parts = [int(p) for p in ts.split(":")]
while len(parts) < 3:
parts.insert(0, 0)
h, m, s = parts
return h * 3600 + m * 60 + s
def fmt(sec, force_hours=False):
sec = int(round(sec))
h, rem = divmod(sec, 3600)
m, s = divmod(rem, 60)
return f"{h:d}:{m:02d}:{s:02d}" if h or force_hours else f"{m:02d}:{s:02d}"
def words(text):
return re.findall(r"[A-Za-z][A-Za-z'-]+", text)
def parse(text):
"""Return a list of units: dict(kind=text|heading|pause, text, time or None, speaker)."""
units, pending_time = [], None
lines = text.replace("\r\n", "\n").replace("\r", "\n").lstrip("\ufeff").split("\n")
para = []
def flush():
if para:
joined = " ".join(p["text"] for p in para)
units.append({"kind": "text", "text": joined, "time": para[0]["time"], "speaker": para[0]["speaker"]})
para.clear()
for raw in lines:
line = raw.strip()
if not line or line.upper() == "WEBVTT" or line.isdigit():
if not line:
flush()
continue
m = CUE_LINE.match(line)
if m:
pending_time = to_sec(m.group(1))
if m.group(2):
LAST_CUE_END[0] = max(LAST_CUE_END[0], to_sec(m.group(2)))
continue
h = HEADING.match(line)
if h:
flush()
units.append({"kind": "heading", "text": (h.group(1) or h.group(2)).strip(), "time": pending_time,
"speaker": None})
continue
if PAUSE.match(line):
flush()
units.append({"kind": "pause", "text": line, "time": pending_time, "speaker": None})
continue
t = TS_LINE.match(line)
time = pending_time
if t and t.group(2) is not None and (":" in t.group(1)):
time, line = to_sec(t.group(1)), t.group(2).strip()
flush()
if not line:
pending_time = time
continue
h = HEADING.match(line)
if h:
units.append({"kind": "heading", "text": (h.group(1) or h.group(2)).strip(), "time": time, "speaker": None})
pending_time = None
continue
if PAUSE.match(line):
units.append({"kind": "pause", "text": line, "time": time, "speaker": None})
pending_time = None
continue
sp = SPEAKER.match(line)
speaker = None
if sp and len(sp.group(1)) <= 30:
speaker, line = sp.group(1), sp.group(2)
flush()
if time is not None or pending_time is not None:
flush()
para.append({"text": line, "time": time, "speaker": speaker})
pending_time = None
flush()
return units
def assign_times(units, wpm, offset):
"""Fill missing times by interpolating between known ones or from the word count."""
known = [i for i, u in enumerate(units) if u["time"] is not None]
clock = 0.0
timed = bool(known)
for i, u in enumerate(units):
if u["time"] is not None:
clock = float(u["time"])
u["estimated"] = False
else:
u["time"] = clock
u["estimated"] = True
clock = u["time"] + len(words(u["text"])) / wpm * 60 if u["kind"] == "text" else u["time"]
for u in units:
u["time"] += offset
return timed
EXTRA_STOP = set()
def keywords(text, n=4):
ws = [w.lower().strip("'-") for w in words(text)]
ws = [w for w in ws if len(w) > 2 and w not in STOP and w not in EXTRA_STOP]
counts = Counter(ws)
bigrams = Counter(f"{a} {b}" for a, b in zip(ws, ws[1:]) if a != b)
out = [b for b, c in bigrams.most_common(3) if c >= 2]
for w, _c in counts.most_common(12):
if len(out) >= n:
break
if not any(w in b.split() for b in out):
out.append(w)
return out[:n]
def similarity(a, b):
wa = {w.lower() for w in words(a) if w.lower() not in STOP and len(w) > 2}
wb = {w.lower() for w in words(b) if w.lower() not in STOP and len(w) > 2}
if not wa or not wb:
return 0.0
return len(wa & wb) / len(wa | wb)
def boundaries(units, min_gap, max_chapters, end_time):
"""Score each unit as a chapter start; return chosen indexes (always includes the first text unit)."""
scores = {}
for i, u in enumerate(units):
if u["kind"] == "heading":
scores[i] = 10.0
elif u["kind"] == "pause":
nxt = next((j for j in range(i + 1, len(units)) if units[j]["kind"] == "text"), None)
if nxt is not None:
scores[nxt] = max(scores.get(nxt, 0), 3.0)
elif u["kind"] == "text":
low = u["text"].lower()
cue = 4.0 if any(c in low[:160] for c in CUES) else 1.5 if any(c in low[:160] for c in ITEM_CUES) else 0.0
before = " ".join(x["text"] for x in units[max(0, i - 3):i] if x["kind"] == "text")
after = " ".join(x["text"] for x in units[i:i + 3] if x["kind"] == "text")
shift = 1.0 - similarity(before, after) if before and after else 0.0
score = shift * 2.0 + cue
if score > 1.6:
scores[i] = max(scores.get(i, 0), score)
first = next((i for i, u in enumerate(units) if u["kind"] in ("text", "heading")), 0)
chosen = [first]
for i in sorted(scores, key=lambda k: (-scores[k], k)):
if len(chosen) >= max_chapters:
break
t = units[i]["time"]
forced = units[i]["kind"] == "heading"
strong = forced or scores[i] >= 4.0 # heading or cue phrase
if end_time - t < (60 if strong else min_gap / 2) and not forced:
continue # too close to the end of the episode
if all(abs(t - units[c]["time"]) >= min_gap for c in chosen) or (forced and i not in chosen and
all(abs(t - units[c]["time"]) >= 10 for c in chosen)):
chosen.append(i)
return sorted(set(chosen)), scores
def sentences(text):
return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]
def quote_candidates(units, n):
cands = []
for u in units:
if u["kind"] != "text":
continue
for s in sentences(u["text"]):
clean = re.sub(r"\s{2,}", " ", FILLER.sub("", s)).strip()
wc = len(clean.split())
if wc < 8 or wc > 40 or clean.endswith("?"):
continue
score = 0.0
score += 2.0 if 12 <= wc <= 28 else 0.5
score += 1.5 * len(QUOTE_WORDS.findall(clean))
score += 1.0 if re.search(r"\b\d+(?:[.,]\d+)?\s*(%|percent|x|times|years|hours|days)\b", clean, re.I) else 0
score += 0.5 if re.search(r"\b(I|we)\b", clean) else 0
score -= 1.0 if FILLER.search(s) else 0
score -= 3.0 if INTRO.search(clean) else 0
cands.append({"time": u["time"], "speaker": u["speaker"], "text": clean, "score": round(score, 1),
"edited": clean != s})
cands.sort(key=lambda c: (-c["score"], c["time"]))
out, seen = [], set()
for c in cands:
key = c["text"][:40].lower()
if key in seen:
continue
seen.add(key)
out.append(c)
if len(out) >= n:
break
return sorted(out, key=lambda c: c["time"])
def build(units, chosen, end_time):
chapters = []
for k, start in enumerate(chosen):
stop = chosen[k + 1] if k + 1 < len(chosen) else len(units)
seg = units[start:stop]
text = " ".join(u["text"] for u in seg if u["kind"] == "text")
heading = next((u["text"] for u in seg[:1] if u["kind"] == "heading"), None)
t0 = units[start]["time"]
t1 = units[stop]["time"] if stop < len(units) else end_time
speakers = Counter(u["speaker"] for u in seg if u["speaker"])
first = sentences(text)[:1]
chapters.append({
"n": k + 1, "start": round(t0), "start_hms": fmt(t0), "end": round(t1), "minutes": round((t1 - t0) / 60, 1),
"estimated_time": any(u.get("estimated") for u in seg[:1]),
"source": "heading" if heading else "detected",
"title_draft": heading or " / ".join(keywords(text)).title() or "(untitled)",
"keywords": keywords(text, 6), "words": len(words(text)),
"speakers": [s for s, _ in speakers.most_common()],
"opening_line": first[0][:160] if first else "",
})
return chapters
def youtube_lines(chapters):
lines, problems = [], []
if chapters and chapters[0]["start"] != 0:
problems.append("YouTube needs the first chapter at 00:00; the first one was moved to 00:00")
for c in chapters:
start = 0 if c is chapters[0] else c["start"]
lines.append(f"{fmt(start, force_hours=chapters[-1]['start'] >= 3600)} {c['title_draft']}")
if len(chapters) < 3:
problems.append("YouTube shows chapters only with at least 3 of them")
short = [c["n"] for c in chapters if c["minutes"] * 60 < 10]
if short:
problems.append(f"chapters {short} are shorter than 10 seconds (YouTube minimum)")
return lines, problems
def parse_args(argv):
opts = {"file": None, "min_chapter": 4.0, "max_chapters": 12, "wpm": 150.0, "offset": 0.0, "quotes": 6,
"format": "text", "duration": None}
numeric = {"--min-chapter": ("min_chapter", float), "--max-chapters": ("max_chapters", int),
"--wpm": ("wpm", float), "--offset": ("offset", float), "--quotes": ("quotes", int)}
i = 0
while i < len(argv):
a = argv[i]
if a in ("-h", "--help"):
print(__doc__)
sys.exit(0)
if a in numeric or a in ("--format", "--duration"):
if i + 1 >= len(argv):
raise UsageError(f"{a} needs a value")
v = argv[i + 1]
if a == "--format":
if v not in ("text", "json", "youtube"):
raise UsageError("--format must be text, json, or youtube")
opts["format"] = v
elif a == "--duration":
if not re.fullmatch(r"(?:\d{1,2}:)?\d{1,2}:\d{2}", v):
raise UsageError("--duration must look like 52:30 or 1:05:00")
opts["duration"] = to_sec(v)
else:
key, typ = numeric[a]
try:
opts[key] = typ(v)
except ValueError:
raise UsageError(f"{a} needs a number, got {v!r}")
if opts[key] < 0 or (key in ("wpm", "max_chapters") and opts[key] == 0):
raise UsageError(f"{a} must be positive")
i += 2
elif a.startswith("--"):
raise UsageError(f"unknown option {a}")
elif opts["file"] is None:
opts["file"] = a
i += 1
else:
raise UsageError(f"unexpected argument {a!r}")
if opts["file"] is None:
raise UsageError("give a transcript file, or - for stdin")
return opts
def main(argv):
try:
opts = parse_args(argv)
if opts["file"] == "-":
text = sys.stdin.read()
else:
with open(opts["file"], encoding="utf-8") as f:
text = f.read()
except UsageError as e:
print(f"error: {e}", file=sys.stderr)
return 2
except (OSError, UnicodeDecodeError) as e:
print(f"error: cannot read {opts['file']}: {getattr(e, 'strerror', None) or e}", file=sys.stderr)
return 2
units = parse(text)
total_words = sum(len(words(u["text"])) for u in units if u["kind"] == "text")
if total_words < 50:
print(f"error: only {total_words} words of transcript; need at least 50 to suggest chapters", file=sys.stderr)
return 1
timed = assign_times(units, opts["wpm"], opts["offset"])
last = units[-1]
end_time = last["time"] + (len(words(last["text"])) / opts["wpm"] * 60 if last["kind"] == "text" else 0)
end_time = max(end_time, LAST_CUE_END[0] + opts["offset"] if LAST_CUE_END[0] else 0)
if opts["duration"]:
end_time = max(end_time, opts["duration"])
EXTRA_STOP.update(w.lower() for u in units if u["speaker"] for w in u["speaker"].split())
chosen, _scores = boundaries(units, opts["min_chapter"] * 60, opts["max_chapters"], end_time)
chapters = build(units, chosen, end_time)
quotes = quote_candidates(units, opts["quotes"])
speakers = Counter(u["speaker"] for u in units if u["speaker"])
notes = []
if not timed:
notes.append(f"no timestamps found: times are estimates at {opts['wpm']:g} words per minute; replace them "
"with real times from the audio editor before publishing")
if opts["duration"] and opts["duration"] < end_time - 30 and timed:
notes.append("the transcript runs past --duration; check the timestamps or the offset")
markers = [{"time": round(u["time"]), "time_hms": fmt(u["time"]), "marker": u["text"]}
for u in units if u["kind"] == "pause"]
yt_lines, yt_problems = youtube_lines(chapters)
report = {"markers": markers, "words": total_words, "timestamps": "found" if timed else "estimated", "end": round(end_time),
"end_hms": fmt(end_time), "speakers": dict(speakers.most_common()), "chapters": chapters,
"quotes": quotes, "youtube": yt_lines, "youtube_problems": yt_problems, "notes": notes}
if opts["format"] == "json":
print(json.dumps(report, indent=2, ensure_ascii=False))
return 0
if opts["format"] == "youtube":
print("\n".join(yt_lines))
for p in yt_problems + notes:
print(f"# note: {p}")
return 0
spk = ", ".join(f"{s} ({c} turns)" for s, c in speakers.most_common()) or "not labeled"
print(f"Transcript: {total_words} words, about {fmt(end_time)} long, timestamps {report['timestamps']}, "
f"speakers: {spk}")
print(f"\nChapter candidates ({len(chapters)}, min {opts['min_chapter']:g} min apart):")
for c in chapters:
est = "~" if c["estimated_time"] else " "
print(f" {c['n']:>2}. {est}{c['start_hms']:>8} {c['minutes']:>5.1f} min [{c['source']}] {c['title_draft']}")
print(f" keywords: {', '.join(c['keywords']) or '-'}")
if c["opening_line"]:
print(f" opens: \"{c['opening_line']}\"")
if markers:
print("\nMarkers (breaks, music, ads): " + ", ".join(f"{m['time_hms']} {m['marker']}" for m in markers))
print(f"\nQuote candidates ({len(quotes)}):")
for q in quotes:
who = f"{q['speaker']}: " if q["speaker"] else ""
edited = " (fillers removed, check against audio)" if q["edited"] else ""
print(f" {fmt(q['time']):>8} {who}\"{q['text']}\"{edited}")
print("\nYouTube chapter lines:")
for line in yt_lines:
print(f" {line}")
for p in yt_problems + notes:
print(f"note: {p}")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))【中文说明】用于根据数据库表结构、慢查询和执行计划评估 SQL 索引方案，重点处理复合索引顺序、覆盖列、可搜索性和冗余风险。使用时粘贴 schema、SQL、EXPLAIN 输出及数据库类型，模型会按等值、排序、范围条件分析建议，并指出低选择性或重复索引的代价。 Reviews a schema and slow SQL queries, proposes composite indexes in the right column order (equality, sort, range) with covering columns and tradeoffs, flags non-sargable filters like functions on columns and leading-wildcard LIKE, and warns about duplicate, redundant, and low-selectivity indexes. Use when a user shares a slow query, EXPLAIN output, or schema and asks "what index do I need?" or "too many indexes?". Includes a tested stdlib Python helper.
---
name: sql-index-recommendation-reviewer
description: Reviews a schema and slow SQL queries, proposes composite indexes in the right column order (equality, sort, range) with covering columns and tradeoffs, flags non-sargable filters like functions on columns and leading-wildcard LIKE, and warns about duplicate, redundant, and low-selectivity indexes. Use when a user shares a slow query, EXPLAIN output, or schema and asks "what index do I need?" or "too many indexes?". Includes a tested stdlib Python helper.
---
# SQL Index Recommendation Reviewer
You help developers make slow queries fast without burying their tables in indexes. You read the schema and the query the way a database engineer would: which columns are compared with equality, which with ranges, what the query sorts by, what it joins on, and which indexes already exist. You propose the smallest set of indexes that serves the workload, in the right column order, and you are honest that only EXPLAIN on real data can confirm it.
## Files in this skill
- `scripts/suggest_indexes.py` - parses CREATE TABLE / CREATE INDEX statements and SELECT, UPDATE, and DELETE queries (subqueries and CTE bodies separately), then prints candidate indexes with DDL, the queries each one serves, covering columns, overlaps, and schema findings (Python 3 standard library only; no database connection)
- `references/indexing-rules.md` - column order, covering indexes, partial and expression indexes, sargability, and over-indexing rules
- `references/explain-reading-guide.md` - how to read EXPLAIN in PostgreSQL, MySQL, and SQLite to confirm or reject a suggestion
- `templates/index-review.md` - the review report to return
- `examples/example-shop-orders-review.md` - a worked review of an online shop's slow queries
## Workflow
### 1. Collect the facts
Ask, or assume and say so:
- The engine and version (PostgreSQL, MySQL/MariaDB, SQLite, SQL Server), and the schema with existing indexes (`\d table`, `SHOW CREATE TABLE`, or the migration files).
- The slow queries exactly as the app sends them, with typical parameter values, how often each runs, and how slow it is.
- Table sizes (rows), how write-heavy each table is, and any EXPLAIN or EXPLAIN ANALYZE output.
- Constraints: can the team build indexes online, disk space, and replication lag concerns.
### 2. Run the helper
```bash
python3 scripts/suggest_indexes.py --schema schema.sql slow-queries.sql
python3 scripts/suggest_indexes.py all-in-one.sql --dialect mysql
python3 scripts/suggest_indexes.py --schema schema.sql - --json < query.sql
python3 scripts/suggest_indexes.py --schema schema.sql queries.sql --strict # exit 1 on non-sargable filters or duplicate indexes
```
The helper is a heuristic parser, not a planner: it does not know row counts, data distribution, or which plan the database will choose. It can miss predicates in complex SQL (it lists them as "not understood"). Read every query yourself and treat each suggestion as a candidate to test. If you cannot run the script, apply `references/indexing-rules.md` by hand in the same order.
### 3. Decide
1. Fix the query before adding an index: rewrite non-sargable filters (functions on columns, arithmetic, leading-wildcard LIKE, OR across columns) so an index can be used at all.
2. For each query, choose column order: equality columns (most selective first when you know it), then ORDER BY or GROUP BY columns if the query sorts with LIMIT, then one range column.
3. Merge: one composite index can serve several queries when they share its leftmost columns. Prefer extending an existing index over adding a new one.
4. Add covering (INCLUDE) columns only for hot, read-heavy queries, and keep them few.
5. Remove before adding: drop duplicate and redundant-prefix indexes after checking usage statistics (`pg_stat_user_indexes`, `sys.schema_unused_indexes`).
6. Count the cost: every index slows inserts and updates of its columns and uses disk and memory. Write-heavy tables deserve fewer, broader indexes.
### 4. Deliver
Fill in `templates/index-review.md`: per-query diagnosis, the recommended DDL in build order with the queries each serves, indexes to drop, query rewrites, tradeoffs, and how to verify with EXPLAIN before and after (see `references/explain-reading-guide.md`). Follow the style of `examples/example-shop-orders-review.md`.
## Rules
- Never claim a speed-up without saying how to measure it; give expected plan changes ("index scan instead of sequential scan and sort"), not invented timings.
- Use safe build commands for production: `CREATE INDEX CONCURRENTLY` in PostgreSQL, online DDL (`ALGORITHM=INPLACE, LOCK=NONE`) in MySQL, and a maintenance window for SQLite.
- Do not recommend an index on a low-selectivity column alone (status, booleans); use it inside a composite or as a partial index condition.
- Keep primary keys and unique constraints; never suggest dropping them to remove "redundancy".
- Tell the user to build one index at a time and compare plans, and to remove indexes that EXPLAIN never uses.
FILE:references/indexing-rules.md
# Indexing rules
The rules the helper applies, plus the judgment it cannot make. Examples use PostgreSQL syntax unless noted.
## 1. Column order in a composite index
A B-tree index is sorted by its first column, then the second, and so on. A query can seek on a leftmost run of columns compared with equality, then use one more column for a range or for sorting.
Order: **equality columns, then sort columns, then one range column**.
```sql
-- WHERE customer_id = ? AND status = 'paid' ORDER BY created_at DESC LIMIT 20
CREATE INDEX ON orders (customer_id, status, created_at);
```
- Among equality columns, put the one used by the most queries first so the index serves them all; selectivity matters less than reuse once all of them are equalities.
- After a range column (`>`, `<`, `BETWEEN`, `LIKE 'abc%'`), later columns cannot be used for seeking, only for filtering inside the index.
- A sort can use the index only if every ORDER BY column comes from the index in the same order and directions are all the same (or all reversed). Mixed directions need a matching index definition (`created_at DESC, id ASC`).
- With a range and an ORDER BY on a different column, the database must choose: seek with the range and sort, or walk the sort order and filter. With LIMIT, walking the sort order often wins.
## 2. Leftmost prefix and merging
An index on `(a, b, c)` serves queries on `a`, on `a, b`, and on `a, b, c`, but not on `b` or `c` alone. Two candidates where one is a leftmost prefix of the other become one index. An existing index `(a)` is redundant once `(a, b)` exists, unless it is unique, partial, or much smaller and very hot.
## 3. Joins and foreign keys
- Index the column on the side that is looked up: usually the foreign key (`orders.customer_id`). The primary key side is already indexed.
- PostgreSQL does not index foreign keys automatically (MySQL InnoDB does). Missing FK indexes also make deletes on the parent table slow.
- A join column plus that table's own filters can form one composite: `(customer_id, created_at)`.
## 4. Covering indexes
If the index holds every column the query reads, the database can skip the table (index-only scan).
- PostgreSQL 11+ and SQL Server: `INCLUDE (col, ...)` stores extra columns without sorting by them.
- MySQL and SQLite: append the columns to the key; InnoDB secondary indexes already carry the primary key.
- Use it for hot, read-mostly queries with few selected columns. Wide covering indexes are big and slow to write.
- PostgreSQL index-only scans also need a well-vacuumed table (visibility map).
## 5. Partial and expression indexes
- Partial (PostgreSQL, SQLite; filtered index in SQL Server): `CREATE INDEX ON orders (created_at) WHERE status = 'paid' AND shipped_at IS NULL;` Tiny and fast when the condition matches few rows. The query must repeat the same condition.
- Expression: `CREATE INDEX ON users (lower(email));` serves `WHERE lower(email) = ?`. MySQL 8 supports functional indexes `((lower(email)))` or a generated column with an index.
## 6. Sargability: filters an index cannot use
| Pattern | Problem | Fix |
|---|---|---|
| `WHERE date(created_at) = '2026-10-12'` | function on the column | `created_at >= '2026-10-12' AND created_at < '2026-10-13'` |
| `WHERE YEAR(updated_at) = 2026` | function on the column | range on the column |
| `WHERE price * 1.2 > 100` | arithmetic on the column | `price > 100 / 1.2` |
| `WHERE name LIKE '%lamp%'` | leading wildcard | full-text search, or `pg_trgm` GIN index |
| `WHERE lower(name) = ?` | function | expression index on `lower(name)` or a case-insensitive collation |
| `WHERE a = 1 OR b = 2` | OR across columns | `UNION` of two indexed queries, or one index per column (bitmap OR in PostgreSQL) |
| `WHERE status <> 'done'`, `NOT IN` | negative filter | positive list `IN (...)` or a partial index |
| `WHERE id = '42'` on an integer column | implicit cast (MySQL may skip the index on string columns compared to numbers) | compare with the right type |
## 7. Over-indexing
- Every index is updated on INSERT, on DELETE, and on UPDATE of its columns; it also uses disk, memory (buffer cache), and backup and replication bandwidth.
- Duplicates (same columns, same condition) are pure cost: drop one.
- Single-column indexes on low-selectivity columns (booleans, status, type) are rarely used; the planner prefers a scan when a value matches more than about 5 to 20 percent of rows. Use them inside composites or as partial-index conditions.
- As a rule of thumb, more than 5 or 6 indexes on a write-heavy table deserves a review; read-mostly tables can carry more.
- Find unused indexes before dropping: PostgreSQL `pg_stat_user_indexes.idx_scan = 0` (since the last stats reset, on every replica), MySQL `sys.schema_unused_indexes`. Drop with `DROP INDEX CONCURRENTLY` in PostgreSQL.
## 8. Building safely
- PostgreSQL: `CREATE INDEX CONCURRENTLY` avoids blocking writes; it cannot run inside a transaction and leaves an INVALID index on failure (drop and retry).
- MySQL 8: `ALTER TABLE ... ADD INDEX ..., ALGORITHM=INPLACE, LOCK=NONE`; watch replication lag on large tables.
- SQLite: index builds lock the database; run during low traffic.
- Build one index at a time, check the plan, then the next.
FILE:references/explain-reading-guide.md
# Reading EXPLAIN to confirm an index
Always compare the plan before and after, on production-like data and with realistic parameter values. Small test tables make the planner choose scans that are right for small tables.
## PostgreSQL
```sql
EXPLAIN (ANALYZE, BUFFERS) SELECT ...; -- runs the query; wrap writes in BEGIN ... ROLLBACK
```
| Plan node | Meaning |
|---|---|
| Seq Scan | reads the whole table; fine for small tables or when most rows match |
| Index Scan | seeks the index, then reads matching table rows |
| Index Only Scan | answered from the index; check "Heap Fetches" is low (vacuum if not) |
| Bitmap Index Scan + Bitmap Heap Scan | collects many matches, then reads pages in order; common for medium selectivity and OR |
| Sort (with "external merge") | sorting spilled to disk; an index in ORDER BY order can remove it |
| Rows Removed by Filter | rows read and thrown away; high numbers mean the index does not cover the filter |
Look at: actual time of the top node, rows estimated versus actual (a 10x gap means stale statistics: run `ANALYZE table`), and shared hit/read buffers. Confirm a new index is used with `pg_stat_user_indexes.idx_scan` after a day of traffic.
## MySQL 8
```sql
EXPLAIN FORMAT=TREE SELECT ...;
EXPLAIN ANALYZE SELECT ...; -- 8.0.18+, runs the query
```
In classic `EXPLAIN` output check: `type` (ALL = full scan, ref/eq_ref/range = index use, index = full index scan), `key` (the index chosen), `rows` (estimate), and `Extra` ("Using index" = covering, "Using filesort" = extra sort, "Using temporary" = temporary table for GROUP BY or DISTINCT). Refresh statistics with `ANALYZE TABLE t`.
## SQLite
```sql
EXPLAIN QUERY PLAN SELECT ...;
```
"SCAN t" is a full scan; "SEARCH t USING INDEX idx (a=? AND b>?)" shows which columns seek; "USING COVERING INDEX" means no table access; "USE TEMP B-TREE FOR ORDER BY" means an extra sort. Run `ANALYZE;` so the planner has statistics.
## What success looks like
- The slow query's plan switches from a full scan plus sort to an index (only) scan with no sort, and the rows read drop close to the rows returned.
- Writes on the table do not get noticeably slower (check p95 insert and update latency after deploying).
- No other important query changed plan for the worse; run the top queries by total time (`pg_stat_statements`, MySQL performance schema digest tables) before and after.
## When the index is not used
1. Statistics are stale: run ANALYZE.
2. The filter is not sargable (see indexing-rules.md section 6).
3. The value matches too many rows; the planner is right to scan.
4. Column types or collations differ between the query and the column (implicit cast).
5. A parameterized query uses a generic plan; test with the real prepared statement.
6. The index columns are in the wrong order for the query.
FILE:templates/index-review.md
# Index review: app_or_service
**Engine:** engine_and_version | **Tables:** tables_with_row_counts | **Write load:** write_profile
**Helper:** n_queries queries analyzed, n_suggested candidates, n_findings schema findings
## Summary
Two or three sentences: the main cause of slowness, the indexes to add and drop, and the expected plan change.
## Per-query diagnosis
| Query | Runs | Now (plan / time) | Problem | Fix |
|---|---|---|---|---|
| Q1 | per_minute | Seq Scan + Sort | | |
## Query rewrites (do these first)
```sql
-- before
-- after
```
## Indexes to add (build in this order)
| # | DDL | Serves | Why this column order | Cost |
|---|---|---|---|---|
| 1 | | | | |
## Indexes to drop (after checking usage)
| Index | Reason | Check first |
|---|---|---|
| | duplicate / redundant prefix / unused / low selectivity | `idx_scan` on primary and replicas |
## Net effect per table
| Table | Indexes before | After | Note |
|---|---|---|---|
## How to verify
1. Capture `EXPLAIN (ANALYZE, BUFFERS)` for each query before.
2. Build one index (`CONCURRENTLY` / online DDL), run `ANALYZE`, capture the plan again.
3. Expect: (plan change per query).
4. Watch write latency and replication lag for a day; then drop the indexes listed above.
## Caveats
- Suggestions come from query text and schema only; row counts and data skew can change the best choice.
FILE:examples/example-shop-orders-review.md
# Example: index review for an online shop
## User request
"Our PostgreSQL 16 shop database is getting slow. orders has 12 million rows and gets about 40 inserts a second, customers has 900k rows, products 20k. Here are the schema and the six queries that show up in pg_stat_statements. What indexes should we add, and do we have too many already?"
## Helper run
```
$ python3 scripts/suggest_indexes.py --schema shop-schema.sql slow-queries.sql
Schema: 4 table(s), 12 existing index(es) incl. keys | dialect: postgres
Queries analyzed: 7
Q1 SELECT: SELECT o.id, o.total, o.created_at FROM orders o WHERE o.customer_id = $1 AND o.status = 'paid' ORDER BY o.cre...
equality: orders.customer_id (=), orders.status (=)
order by: orders.created_at DESC (LIMIT)
Q2 SELECT: SELECT c.email, count(*) AS orders FROM customers c JOIN orders o ON o.customer_id = c.id WHERE c.country = 'D...
equality: customers.country (=)
range: orders.created_at (range)
joins: orders.customer_id = customers.id
group by: customers.email
Q3 SELECT: SELECT * FROM products WHERE lower(name) LIKE '%lamp%'
WARN non-sargable: lower(name) in the WHERE clause hides the column from a normal index; ... the LIKE pattern '%lamp%' also starts with a wildcard, so only full-text search or a trigram index can help
Q4 SELECT: ... WHERE oi.order_id IN (SELECT id FROM orders WHERE customer_id = 42 AND created_at > now() - interval '30 days')
note: subquery analyzed as Q5
Q6 UPDATE: UPDATE orders SET status = 'shipped', shipped_at = now() WHERE status = 'paid' AND shipped_at IS NULL AND crea...
Q7 SELECT: SELECT id FROM customers WHERE email = 'ann@example.com' OR phone = '+491701234567'
WARN OR across different columns (...): one composite index cannot serve both sides; consider UNION ALL of two indexed queries or one index per side
Suggested indexes (4):
1. CREATE INDEX CONCURRENTLY idx_customers_country_email ON customers (country, email);
2. CREATE INDEX CONCURRENTLY idx_orders_customer_id_created_at ON orders (customer_id, created_at);
serves: Q2, Q5 | overlap: same leading column as #3
3. CREATE INDEX CONCURRENTLY idx_orders_customer_id_status_created_at ON orders (customer_id, status, created_at) INCLUDE (id, total);
serves: Q1 | overlap: same leading column as #2
4. CREATE INDEX CONCURRENTLY idx_orders_status_shipped_at_created_at ON orders (status, shipped_at, created_at);
serves: Q6 | alternative: ... ON orders (status, created_at) WHERE shipped_at IS NULL;
Schema findings (5):
WARN redundant-prefix customers: idx_customers_country (country) is a leftmost prefix of idx_customers_country_created ...
WARN low-selectivity customers: idx_customers_active indexes only 'is_active' ...
HIGH duplicate-index orders: idx_orders_created and orders_created_at_idx have the same columns (created_at); drop one
WARN low-selectivity orders: idx_orders_status indexes only 'status' ...
WARN many-indexes orders: 4 existing + 3 suggested = 7 indexes (limit 6) ...
```
## Review (filled template, shortened)
**Summary:** orders has no index that starts with customer_id, so the order history page (Q1) and the recent-orders lookup (Q5) scan millions of rows, and two of its four indexes are a duplicate and a low-selectivity status index. Add one composite and one partial index on orders, a phone index and a trigram index elsewhere, and drop three indexes: orders ends with the same count of 4 but every one is useful.
| Query | Problem | Fix |
|---|---|---|
| Q1 order history | no index on customer_id; sort on created_at | (customer_id, created_at) with status and total included: newest rows for the customer, status filtered inside the index, LIMIT stops after 20 |
| Q2 country report | join needs orders.customer_id; customers.country is already served by idx_customers_country_created | same orders index; no new customers index (the helper's #1 only helps the GROUP BY, and the report runs once a night) |
| Q3 product search | `lower(name) LIKE '%lamp%'` cannot use a B-tree | pg_trgm GIN index on lower(name) |
| Q4/Q5 recent items | subquery needs (customer_id, created_at); order_items PK (order_id, product_id) serves the IN | same orders index |
| Q6 shipping job | scans by status | partial index on created_at for paid, unshipped orders only |
| Q7 support lookup | OR across email and phone | UNION of two lookups; email is unique already, add a phone index |
Why not the helper's #3 as well: #2 and #3 share customer_id, and on a table with 40 inserts a second one index is better than two. The user confirmed that about 95 percent of orders are paid, so Q1 throws away few rows, and (customer_id, created_at) with status in INCLUDE keeps Q1 fast. If the share of non-paid orders grows, switch to #3's column order.
**Indexes to add (in this order):**
```sql
CREATE INDEX CONCURRENTLY idx_orders_customer_created
ON orders (customer_id, created_at) INCLUDE (id, status, total); -- Q1, Q2, Q5
CREATE INDEX CONCURRENTLY idx_orders_paid_unshipped
ON orders (created_at) WHERE status = 'paid' AND shipped_at IS NULL; -- Q6, holds only paid, unshipped orders
CREATE INDEX CONCURRENTLY idx_customers_phone ON customers (phone); -- Q7 after the rewrite
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX CONCURRENTLY idx_products_name_trgm
ON products USING gin (lower(name) gin_trgm_ops); -- Q3, query unchanged
```
**Query rewrite (Q7):**
```sql
SELECT id FROM customers WHERE email = 'ann@example.com'
UNION
SELECT id FROM customers WHERE phone = '+491701234567';
```
**Indexes to drop (after checking `pg_stat_user_indexes.idx_scan` on the primary and both replicas):**
| Index | Reason |
|---|---|
| orders_created_at_idx | duplicate of idx_orders_created |
| idx_orders_status | low selectivity; Q6 now uses the partial index |
| idx_customers_country | leftmost prefix of idx_customers_country_created |
| idx_customers_active (check) | boolean; keep only if idx_scan shows real use |
**Net effect:** orders 4 -> 4 indexes (two replaced by better ones), customers 5 -> 5 (one dropped, phone added; 4 if is_active goes too), products 2 -> 3.
**Recheck with the helper** (schema after the change, queries after the rewrite; query numbers follow the new file, so Q3 is the old Q5 and Q4/Q5 are the two UNION branches):
```
$ python3 scripts/suggest_indexes.py --schema shop-schema-after.sql shop-queries-after.sql --no-covering
Already served by existing indexes (5):
customers (email) -> customers_email_key (Q4)
customers (id, country) -> customers_pkey (Q2)
customers (phone) -> idx_customers_phone (Q5)
orders (created_at) -> idx_orders_created (Q2)
orders (customer_id, created_at) -> idx_orders_customer_created (Q2, Q3)
```
The helper still lists (customer_id, status, created_at) for Q1 and (country, email) for Q2; both were rejected on purpose for the reasons above. There are no duplicate or redundant-prefix findings left.
**How to verify:** before and after each build, run `EXPLAIN (ANALYZE, BUFFERS)` for Q1 with a real customer id. Expect "Index Scan using idx_orders_customer_created" (or Index Only Scan) with no Sort node and about 20 rows read instead of a Bitmap Heap Scan over thousands. Watch insert latency on orders for a day before dropping the old indexes.
FILE:scripts/suggest_indexes.py
#!/usr/bin/env python3
"""Suggest candidate indexes for slow SQL queries from a schema and the query text (no database needed).
Usage:
python3 suggest_indexes.py --schema schema.sql queries.sql [options]
python3 suggest_indexes.py all-in-one.sql (CREATE statements and queries in one file)
python3 suggest_indexes.py --schema schema.sql - < slow.sql
Options:
--schema FILE CREATE TABLE / CREATE INDEX statements (may also be inside the query file)
--dialect NAME postgres (default), mysql, or sqlite; changes INCLUDE and online-build hints
--max-indexes N warn when a table would have more than N indexes (default 6)
--no-covering do not suggest covering (INCLUDE) columns
--json print the result as JSON
--strict exit 1 if any query has a non-sargable predicate or the schema has duplicate indexes
What it does: lightweight parsing of SELECT, UPDATE, and DELETE statements (subqueries are analyzed
separately) to find equality filters, range filters, join columns, ORDER BY and GROUP BY columns,
and selected columns per table. It builds composite candidates (equality columns, then sort, then
range), merges candidates that are prefixes of each other, skips those already served by an existing
index, and reports duplicate and redundant indexes, low-selectivity single-column indexes, and tables
with many indexes. It does not know row counts or data distribution: confirm every suggestion with
EXPLAIN on real data.
Exit codes: 0 ok, 1 --strict findings, 2 usage or read error.
Python 3 standard library only.
"""
import json
import re
import signal
import sys
if hasattr(signal, "SIGPIPE"):
signal.signal(signal.SIGPIPE, signal.SIG_DFL) # quiet exit when piped into head
KEYWORDS = set("""select from where and or not in is null like ilike between join inner left right full outer cross on
using group by order having limit offset as asc desc distinct union all exists case when then else end update set
delete insert into values returning with true false interval current_date current_timestamp now lateral natural
fetch first rows only for share nowait skip locked""".split())
LOW_CARD = re.compile(r"^(is_|has_|can_|should_)|^(status|state|type|kind|flag|active|enabled|deleted|archived|gender|"
r"visible|published|verified|priority|level|tier|role)$", re.I)
CLAUSE_END = r"(?=\b(?:where|group\s+by|order\s+by|having|limit|offset|union|returning|for\s+update|window)\b|$)"
class UsageError(Exception):
pass
# ---------------------------------------------------------------- text helpers
def strip_comments(sql):
sql = re.sub(r"/\*.*?\*/", " ", sql, flags=re.S)
return re.sub(r"--[^\n]*", " ", sql)
def split_statements(sql):
out, buf, quote = [], [], None
for ch in sql:
if quote:
buf.append(ch)
if ch == quote:
quote = None
continue
if ch in "'\"`":
quote = ch
if ch == ";":
out.append("".join(buf))
buf = []
else:
buf.append(ch)
out.append("".join(buf))
return [s.strip() for s in out if s.strip()]
def unquote_idents(sql):
sql = re.sub(r'"([A-Za-z_][\w$]*)"', r"\1", sql)
sql = re.sub(r"`([A-Za-z_][\w$]*)`", r"\1", sql)
return re.sub(r"\[([A-Za-z_][\w$]*)\]", r"\1", sql)
def mask_strings(sql):
"""Replace string literals with placeholders, remembering LIKE patterns."""
lits = []
def rep(m):
lits.append(m.group(0)[1:-1])
return f" __lit{len(lits) - 1}__ "
return re.sub(r"'(?:[^']|'')*'", rep, sql), lits
def split_top(text, sep_regex):
"""Split on a keyword at parenthesis depth 0."""
parts, depth, last = [], 0, 0
for m in re.finditer(r"\(|\)|" + sep_regex, text, flags=re.I):
tok = m.group(0)
if tok == "(":
depth += 1
elif tok == ")":
depth -= 1
elif depth == 0:
parts.append(text[last:m.start()])
last = m.end()
parts.append(text[last:])
return [p.strip() for p in parts if p.strip()]
def split_and(text):
"""Split on top-level AND, keeping 'x BETWEEN a AND b' together."""
parts = split_top(text, r"\band\b")
out = []
for part in parts:
if out and re.search(r"\bbetween\s+[^\s]+$", out[-1], re.I):
out[-1] = f"{out[-1]} AND {part}"
else:
out.append(part)
return out
def extract_subqueries(sql):
"""Pull out innermost (SELECT ...) blocks; return (outer_sql, [subqueries])."""
subs = []
pattern = re.compile(r"\(\s*(select\b[^()]*(?:\([^()]*\)[^()]*)*)\)", re.I)
while True:
m = pattern.search(sql)
if not m:
break
subs.append(m.group(1))
sql = sql[:m.start()] + " __subq__ " + sql[m.end():]
return sql, subs
# ---------------------------------------------------------------- schema
def parse_columns_list(text):
return [re.sub(r"\s+(asc|desc)\b.*$", "", c.strip(), flags=re.I).strip().lower()
for c in split_top(text, r",")]
def parse_schema(statements):
tables, indexes = {}, []
for st in statements:
s = unquote_idents(strip_comments(st)).strip()
m = re.match(r"create\s+table\s+(?:if\s+not\s+exists\s+)?([\w.]+)\s*\((.*)\)\s*[^)]*$", s, re.I | re.S)
if m:
name = m.group(1).split(".")[-1].lower()
cols, pk = [], None
for part in split_top(m.group(2), r","):
p = part.strip()
low = p.lower()
c = re.match(r"(?:constraint\s+\w+\s+)?primary\s+key\s*\(([^)]*)\)", low)
if c:
pk = parse_columns_list(c.group(1))
continue
c = re.match(r"(?:constraint\s+(\w+)\s+)?unique\s*(?:key\s+\w+\s*)?\(([^)]*)\)", low)
if c:
indexes.append({"name": c.group(1) or f"{name}_unique", "table": name,
"columns": parse_columns_list(c.group(2)), "unique": True, "kind": "unique"})
continue
c = re.match(r"(?:key|index)\s+(\w+)\s*\(([^)]*)\)", low)
if c:
indexes.append({"name": c.group(1), "table": name, "columns": parse_columns_list(c.group(2)),
"unique": False, "kind": "index"})
continue
if re.match(r"(constraint|foreign\s+key|check|exclude)\b", low):
continue
col = re.match(r"([a-z_][\w$]*)\s+", low + " ")
if col:
cols.append(col.group(1))
if re.search(r"\bprimary\s+key\b", low):
pk = [col.group(1)]
elif re.search(r"\bunique\b", low):
indexes.append({"name": f"{name}_{col.group(1)}_key", "table": name,
"columns": [col.group(1)], "unique": True, "kind": "unique"})
tables[name] = {"columns": cols}
if pk:
indexes.append({"name": f"{name}_pkey", "table": name, "columns": pk, "unique": True,
"kind": "primary"})
continue
m = re.match(r"create\s+(unique\s+)?index\s+(?:concurrently\s+)?(?:if\s+not\s+exists\s+)?(\w+)\s+on\s+"
r"([\w.]+)\s*(?:using\s+\w+\s*)?\(([^;]*?)\)\s*(include\s*\(([^)]*)\))?\s*(where\s+.*)?$",
s, re.I | re.S)
if m:
cols = parse_columns_list(m.group(4))
indexes.append({"name": m.group(2), "table": m.group(3).split(".")[-1].lower(), "columns": cols,
"unique": bool(m.group(1)), "kind": "index",
"include": parse_columns_list(m.group(6)) if m.group(6) else [],
"partial": (m.group(7) or "").strip() or None})
return tables, indexes
# ---------------------------------------------------------------- queries
def parse_from(from_text):
"""Return (tables {alias: table}, order list, join predicates text list)."""
aliases, order, ons = {}, [], []
pieces = re.split(r"\b((?:natural\s+)?(?:inner\s+|left\s+(?:outer\s+)?|right\s+(?:outer\s+)?|full\s+(?:outer\s+)?|"
r"cross\s+)?join)\b", from_text, flags=re.I)
items = [pieces[0]] + [pieces[i + 1] for i in range(1, len(pieces) - 1, 2)]
for item in items:
on_split = re.split(r"\bon\b", item, maxsplit=1, flags=re.I)
using = re.search(r"\busing\s*\(([^)]*)\)", on_split[0], re.I)
head = re.sub(r"\busing\s*\([^)]*\)", "", on_split[0], flags=re.I)
for ref in head.split(","):
toks = [t for t in re.split(r"\s+", ref.strip()) if t and t.lower() not in ("as", "lateral", "only")]
if not toks or toks[0] == "__subq__":
continue
table = toks[0].split(".")[-1].lower()
alias = toks[1].lower() if len(toks) > 1 else table
aliases[alias] = table
order.append(alias)
if len(on_split) > 1:
ons.append(on_split[1])
if using and len(order) >= 2:
for col in parse_columns_list(using.group(1)):
ons.append(f"{order[-2]}.{col} = {order[-1]}.{col}")
return aliases, order, ons
COL = r"((?:[a-z_][\w$]*\.)?[a-z_][\w$]*)"
VALUE = r"(?:__lit\d+__|\?|:\w+|\$\d+|@\w+|%s|-?\d+(?:\.\d+)?|true|false|null|__subq__|current_date|current_timestamp|now\(\))"
def resolve(ref, aliases, tables):
ref = ref.lower()
if "." in ref:
a, c = ref.split(".", 1)
return (aliases.get(a, a), c)
if len(set(aliases.values())) == 1:
return (next(iter(aliases.values())), ref)
owners = [t for t in set(aliases.values()) if ref in tables.get(t, {}).get("columns", [])]
return (owners[0], ref) if len(owners) == 1 else (None, ref)
def analyze_predicate(pred, lits, aliases, tables, q):
p = pred.strip()
while p.startswith("(") and p.endswith(")") and len(split_top(p[1:-1], r"\band\b")) >= 1 and \
p.count("(") == p.count(")") and _balanced_wrap(p):
p = p[1:-1].strip()
if len(split_and(p)) > 1:
for sub in split_and(p):
analyze_predicate(sub, lits, aliases, tables, q)
return
ors = split_top(p, r"\bor\b")
if len(ors) > 1:
cols = set()
for o in ors:
m = re.match(rf"^\(?\s*{COL}\s*(=|in\b)", o.strip(), re.I)
cols.add(m.group(1).lower() if m else None)
if len(cols) == 1 and None not in cols:
t, c = resolve(cols.pop(), aliases, tables)
q["filters"].append({"table": t, "column": c, "kind": "eq", "op": "OR of equalities (same as IN)"})
else:
shown = re.sub(r"__lit(\d+)__", lambda m: "'" + lits[int(m.group(1))] + "'", p)
shown = shown if len(shown) <= 70 else shown[:67] + "..."
q["warnings"].append(f"OR across different columns ({shown}): one composite index cannot serve "
"both sides; consider UNION ALL of two indexed queries or one index per side")
return
if re.match(r"^not\b", p, re.I) or re.search(r"(<>|!=|\bnot\s+in\b|\bnot\s+like\b)", p, re.I):
m = re.match(rf"^(?:not\s+)?{COL}", p, re.I)
shown = re.sub(r"__lit(\d+)__", lambda m: "'" + lits[int(m.group(1))] + "'", p)[:50]
q["notes"].append(f"negative predicate '{shown}' rarely uses an index; it is applied as a filter")
return
m = re.match(rf"^{COL}\s*(=|<=>)\s*{COL}$", p, re.I)
if m and not re.fullmatch(VALUE, m.group(3), re.I) and m.group(3).lower() not in KEYWORDS:
left, right = resolve(m.group(1), aliases, tables), resolve(m.group(3), aliases, tables)
if left[0] and right[0] and left[0] != right[0]:
q["joins"].append((left, right))
return
m = re.match(rf"^{VALUE}\s*(=|<|>|<=|>=)\s*{COL}$", p, re.I)
if m:
flip = {"<": ">", ">": "<", "<=": ">=", ">=": "<=", "=": "="}
p = f"{m.group(2)} {flip[m.group(1)]} __v__"
m = re.match(rf"^{COL}\s*(=|<=>)\s*(.+)$", p, re.I)
if m and m.group(1).lower() not in KEYWORDS:
t, c = resolve(m.group(1), aliases, tables)
q["filters"].append({"table": t, "column": c, "kind": "eq", "op": "="})
return
m = re.match(rf"^{COL}\s+in\s+__subq__$", p, re.I)
if m:
t, c = resolve(m.group(1), aliases, tables)
q["filters"].append({"table": t, "column": c, "kind": "eq", "op": "IN (subquery)"})
return
m = re.match(rf"^{COL}\s+in\s*\((.*)\)$", p, re.I | re.S)
if m:
t, c = resolve(m.group(1), aliases, tables)
q["filters"].append({"table": t, "column": c, "kind": "eq", "op": "IN"})
return
m = re.match(rf"^{COL}\s+is\s+(not\s+)?null$", p, re.I)
if m:
t, c = resolve(m.group(1), aliases, tables)
if m.group(2):
q["notes"].append(f"'{c} IS NOT NULL' is usually not selective; a partial index can help if most rows are NULL")
else:
q["filters"].append({"table": t, "column": c, "kind": "eq", "op": "IS NULL"})
return
m = re.match(rf"^{COL}\s*(<|>|<=|>=)\s*(.+)$", p, re.I) or re.match(rf"^{COL}\s+between\s+(.+)\s+and\s+(.+)$", p, re.I)
if m and m.group(1).lower() not in KEYWORDS:
t, c = resolve(m.group(1), aliases, tables)
q["filters"].append({"table": t, "column": c, "kind": "range", "op": "range"})
return
m = re.match(rf"^{COL}\s+(i?like)\s+__lit(\d+)__", p, re.I)
if m:
t, c = resolve(m.group(1), aliases, tables)
pattern = lits[int(m.group(3))]
if pattern.startswith(("%", "_")):
q["warnings"].append(f"non-sargable: {c} {m.group(2).upper()} '{pattern}' starts with a wildcard, so a "
"B-tree index cannot seek; use full-text search or a trigram index (PostgreSQL pg_trgm)")
q["non_sargable"] = True
else:
q["filters"].append({"table": t, "column": c, "kind": "range", "op": f"LIKE '{pattern}'"})
if m.group(2).lower() == "ilike":
q["notes"].append(f"ILIKE on {c}: a plain B-tree index is case-sensitive; index lower({c}) instead")
return
m = re.match(rf"^([a-z_]+)\s*\(\s*{COL}[^)]*\)\s*(=|<|>|<=|>=|like|between)", p, re.I)
if m and m.group(1).lower() not in KEYWORDS:
func, col = m.group(1).lower(), m.group(2)
t, c = resolve(col, aliases, tables)
wild = re.search(r"like\s+__lit(\d+)__", p, re.I)
extra = ""
if wild and lits[int(wild.group(1))].startswith(("%", "_")):
extra = (f"; the LIKE pattern '{lits[int(wild.group(1))]}' also starts with a wildcard, so only full-text "
"search or a trigram index can help")
q["warnings"].append(f"non-sargable: {func}({c}) in the WHERE clause hides the column from a normal index; "
f"rewrite as a range on {c} or add an expression index on {func}({c}){extra}")
q["non_sargable"] = True
return
m = re.match(rf"^{COL}\s*[-+*/]\s*\S+\s*(=|<|>|<=|>=)", p, re.I)
if m:
t, c = resolve(m.group(1), aliases, tables)
q["warnings"].append(f"non-sargable: arithmetic on {c} ('{p[:40]}'); move the arithmetic to the other side")
q["non_sargable"] = True
return
if "__subq__" in p and re.search(r"\bexists\b", p, re.I):
return
q["notes"].append(f"predicate not understood, check by hand: {p[:60]}")
def _balanced_wrap(p):
depth = 0
for i, ch in enumerate(p):
depth += ch == "("
depth -= ch == ")"
if depth == 0 and i < len(p) - 1:
return False
return True
def parse_query(raw, tables, number, lits=None, outer_aliases=None):
if lits is None:
sql, lits = mask_strings(unquote_idents(strip_comments(raw)))
else:
sql = raw
sql = re.sub(r"\s+", " ", sql).strip()
outer, subs = extract_subqueries(sql)
shown = re.sub(r"__lit(\d+)__", lambda m: "'" + lits[int(m.group(1))] + "'", re.sub(r"\s+", " ", raw.strip()))
shown = re.sub(r"\s+", " ", shown)
q = {"n": number, "text": shown[:110] + ("..." if len(shown) > 110 else ""), "kind": None, "aliases": {},
"filters": [], "joins": [], "order_by": [], "group_by": [], "selected": {}, "select_star": False,
"limit": False, "warnings": [], "notes": [], "non_sargable": False, "subqueries": subs, "lits": lits}
low = outer.lower()
cte_names = set()
if low.startswith("with "):
cte_names = {n.lower() for n in re.findall(r"(?:with|,)\s*(?:recursive\s+)?(\w+)\s+as\b", outer, re.I)}
q["notes"].append(f"CTE {', '.join(sorted(cte_names))} analyzed as separate queries; the final SELECT reads from it")
m = re.search(r"(?:\)|__subq__)\s*((?:select|update|delete)\b.*)$", outer, re.I)
outer = m.group(1) if m else outer
low = outer.lower()
if low.startswith("select"):
q["kind"] = "SELECT"
m = re.match(r"select\s+(?:distinct\s+)?(.*?)\s+from\s+(.*?)" + CLAUSE_END, outer, re.I)
if not m:
q["notes"].append("no FROM clause found")
return q
select_list, from_text = m.group(1), m.group(2)
elif low.startswith("update"):
q["kind"] = "UPDATE"
m = re.match(r"update\s+(.*?)\s+set\s+", outer, re.I)
select_list, from_text = "", m.group(1) if m else ""
elif low.startswith("delete"):
q["kind"] = "DELETE"
m = re.match(r"delete\s+from\s+(.*?)" + CLAUSE_END, outer, re.I)
select_list, from_text = "", m.group(1) if m else ""
else:
return None
own, _order, ons = parse_from(from_text)
own = {a: t for a, t in own.items() if t not in cte_names}
q["aliases"] = own
aliases = {**(outer_aliases or {}), **own} # outer aliases resolve correlated references
unknown = [t for t in set(own.values()) if tables and t not in tables]
if unknown:
q["notes"].append(f"table(s) not in the schema: {', '.join(sorted(unknown))}; existing indexes unknown")
for on in ons:
for pred in split_and(on):
analyze_predicate(pred, lits, aliases, tables, q)
w = re.search(r"\bwhere\b(.*?)(?=\b(?:group\s+by|order\s+by|having|limit|offset|returning|for\s+update)\b|$)", outer, re.I)
if w:
for pred in split_and(w.group(1)):
analyze_predicate(pred, lits, aliases, tables, q)
for clause, key in (("group\\s+by", "group_by"), ("order\\s+by", "order_by")):
m = re.search(rf"\b{clause}\b(.*?)(?=\b(?:having|order\s+by|limit|offset|for\s+update)\b|$)", outer, re.I)
if m:
for item in split_top(m.group(1), r","):
mm = re.match(rf"^{COL}\s*(asc|desc)?(?:\s+nulls\s+(?:first|last))?$", item.strip(), re.I)
if mm:
t, c = resolve(mm.group(1), aliases, tables)
q[key].append({"table": t, "column": c, "dir": (mm.group(2) or "asc").lower()})
else:
q["notes"].append(f"{key.replace('_', ' ')} expression '{item.strip()[:40]}' cannot use a plain column index")
q["limit"] = bool(re.search(r"\b(limit|fetch\s+first)\b", outer, re.I))
if select_list:
if re.search(r"(^|,)\s*(\w+\.)?\*\s*(,|$)", select_list):
q["select_star"] = True
for ref in re.findall(rf"\b{COL}\b(?!\s*\()", select_list, re.I):
if ref.lower() in KEYWORDS or ref.startswith("__") or re.fullmatch(r"\d+", ref):
continue
t, c = resolve(ref, aliases, tables)
if t and (not tables or c in tables.get(t, {}).get("columns", [c])):
q["selected"].setdefault(t, [])
if c not in q["selected"][t]:
q["selected"][t].append(c)
for f in q["filters"]:
if f["table"] is None:
q["notes"].append(f"column '{f['column']}' could belong to several tables; qualify it with an alias")
return q
# ---------------------------------------------------------------- candidates
def candidates_for(q, dialect, covering, pk_cols=None):
out = []
by_table = {}
for f in q["filters"]:
if f["table"]:
by_table.setdefault(f["table"], {"eq": [], "range": []})
lst = by_table[f["table"]][f["kind"]]
if f["column"] not in lst:
lst.append(f["column"])
tables_in_q = set(q["aliases"].values())
correlated = set()
for left, right in q["joins"]:
if left[0] in tables_in_q and right[0] not in tables_in_q:
correlated.add(left[0])
if right[0] in tables_in_q and left[0] not in tables_in_q:
correlated.add(right[0])
order_tables = {o["table"] for o in q["order_by"]}
for t in tables_in_q:
if t in correlated:
continue # reached through the correlated join below
eq = by_table.get(t, {}).get("eq", [])
rng = [c for c in by_table.get(t, {}).get("range", []) if c not in eq]
sort = []
if q["order_by"] and order_tables == {t} and len({o["dir"] for o in q["order_by"]}) == 1:
sort = [o["column"] for o in q["order_by"] if o["column"] not in eq]
elif q["group_by"] and {g["table"] for g in q["group_by"]} == {t}:
sort = [g["column"] for g in q["group_by"] if g["column"] not in eq]
key, why = list(eq), []
if eq:
why.append(f"equality on {', '.join(eq)} first")
if sort and (not rng or q["limit"]):
key += [c for c in sort if c not in key]
purpose = "already sorted" if q["order_by"] else "already grouped"
why.append(f"then {', '.join(sort)} so rows come out {purpose}" + (" (stops early with LIMIT)" if q["limit"] else ""))
if rng and rng[0] not in key:
key.append(rng[0])
why.append(f"{rng[0]} last as an in-index filter")
elif rng:
key.append(rng[0])
why.append(f"then the range on {rng[0]}" if eq else f"range on {rng[0]}")
if len(rng) > 1:
q["notes"].append(f"several range filters on {t} ({', '.join(rng)}): only the first can seek; the rest filter")
if sort:
q["notes"].append(f"ORDER BY on {t} cannot use the same index as the range on {rng[0]}; expect a sort step")
if not key:
continue
include = []
if covering and not q["select_star"] and q["kind"] == "SELECT":
extra = [c for c in q["selected"].get(t, []) if c not in key
and not (dialect == "mysql" and c in (pk_cols or {}).get(t, []))]
if 0 < len(extra) <= 3:
include = extra
out.append({"table": t, "columns": key, "include": include, "why": "; ".join(why), "queries": [q["n"]],
"source": "filter", "eq": list(eq),
"null_filter": [f["column"] for f in q["filters"] if f["table"] == t and f["op"] == "IS NULL"]})
for left, right in q["joins"]:
for (t, c), (ot, oc) in ((left, right), (right, left)):
if t not in tables_in_q:
continue # outer table of a correlated subquery
eq = [e for e in by_table.get(t, {}).get("eq", []) if e != c]
rng = [r for r in by_table.get(t, {}).get("range", []) if r != c and r not in eq][:1]
why = f"join column {t}.{c} = {ot}.{oc} first (lets the planner look up matching rows)"
if eq or rng:
why += f", then {', '.join(eq + rng)} to filter inside the index"
if t in correlated:
why = f"correlated lookup {t}.{c} = outer {ot}.{oc} first" + (f", then {', '.join(eq + rng)}" if eq or rng else "")
out.append({"table": t, "columns": [c] + eq + rng, "include": [], "queries": [q["n"]], "source": "join",
"eq": [c] + eq, "why": why})
return out
def served_by(cand, indexes):
for ix in indexes:
if ix["table"] != cand["table"]:
continue
cols = ix["columns"]
if cand["source"] == "join" and ix["unique"] and cols == cand["columns"][:1] and not ix.get("partial"):
return ix # joining on a unique key: one row per lookup, extra columns add little
if cols[:len(cand["columns"])] == cand["columns"]:
return ix
n_eq = len(cand.get("eq", []))
if n_eq > 1 and len(cols) >= len(cand["columns"]) and set(cols[:n_eq]) == set(cand["columns"][:n_eq]) \
and cols[n_eq:len(cand["columns"])] == cand["columns"][n_eq:]:
return ix # same equality columns in another order serve the query equally well
return None
def served_by_named(c, indexes):
ix = served_by(c, indexes)
return {"existing": ix["name"] if ix else "?"}
def merge(cands):
merged = []
for c in sorted(cands, key=lambda c: -len(c["columns"])):
for m in merged:
if m["table"] == c["table"] and m["columns"][:len(c["columns"])] == c["columns"]:
m["queries"] = sorted(set(m["queries"]) | set(c["queries"]))
m["include"] = m["include"] or []
break
else:
merged.append(dict(c))
return sorted(merged, key=lambda c: (c["table"], c["columns"]))
def schema_findings(indexes, max_indexes, new_by_table):
out = []
by_table = {}
for ix in indexes:
by_table.setdefault(ix["table"], []).append(ix)
for t, ixs in by_table.items():
for i, a in enumerate(ixs):
for b in ixs[i + 1:]:
if a["columns"] == b["columns"] and (a.get("partial") == b.get("partial")):
out.append({"level": "high", "rule": "duplicate-index",
"message": f"{t}: {a['name']} and {b['name']} have the same columns ({', '.join(a['columns'])}); drop one"})
elif a["columns"] != b["columns"]:
short, long_ = (a, b) if len(a["columns"]) < len(b["columns"]) else (b, a)
if long_["columns"][:len(short["columns"])] == short["columns"] and not short["unique"] \
and not short.get("partial"):
out.append({"level": "warn", "rule": "redundant-prefix",
"message": f"{t}: {short['name']} ({', '.join(short['columns'])}) is a leftmost prefix of "
f"{long_['name']} ({', '.join(long_['columns'])}); usually droppable"})
for ix in ixs:
if len(ix["columns"]) == 1 and LOW_CARD.search(ix["columns"][0]) and ix["kind"] == "index" and not ix.get("partial"):
out.append({"level": "warn", "rule": "low-selectivity",
"message": f"{t}: {ix['name']} indexes only '{ix['columns'][0]}', which sounds like a flag or status "
"with few values; use a composite or partial index instead"})
total = len(ixs) + new_by_table.get(t, 0)
if total > max_indexes:
out.append({"level": "warn", "rule": "many-indexes",
"message": f"{t}: {len(ixs)} existing + {new_by_table.get(t, 0)} suggested = {total} indexes "
f"(limit {max_indexes}); every insert and update maintains all of them"})
return out
def ddl(c, dialect):
name = f"idx_{c['table']}_{'_'.join(c['columns'])}"[:63]
cols = ", ".join(c["columns"])
if c["include"]:
if dialect == "postgres":
return f"CREATE INDEX CONCURRENTLY {name} ON {c['table']} ({cols}) INCLUDE ({', '.join(c['include'])});"
cols += ", " + ", ".join(c["include"])
if dialect == "postgres":
return f"CREATE INDEX CONCURRENTLY {name} ON {c['table']} ({cols});"
if dialect == "mysql":
return f"ALTER TABLE {c['table']} ADD INDEX {name} ({cols}), ALGORITHM=INPLACE, LOCK=NONE;"
return f"CREATE INDEX {name} ON {c['table']} ({cols});"
# ---------------------------------------------------------------- main
def parse_args(argv):
opts = {"schema": None, "files": [], "dialect": "postgres", "max_indexes": 6, "covering": True,
"json": False, "strict": False}
i = 0
while i < len(argv):
a = argv[i]
if a in ("-h", "--help"):
print(__doc__)
sys.exit(0)
if a in ("--schema", "--dialect", "--max-indexes"):
if i + 1 >= len(argv):
raise UsageError(f"{a} needs a value")
v = argv[i + 1]
if a == "--schema":
opts["schema"] = v
elif a == "--dialect":
if v not in ("postgres", "mysql", "sqlite"):
raise UsageError("--dialect must be postgres, mysql, or sqlite")
opts["dialect"] = v
else:
try:
opts["max_indexes"] = int(v)
except ValueError:
raise UsageError(f"--max-indexes needs a whole number, got {v!r}")
i += 2
elif a == "--no-covering":
opts["covering"] = False; i += 1
elif a == "--json":
opts["json"] = True; i += 1
elif a == "--strict":
opts["strict"] = True; i += 1
elif a.startswith("--"):
raise UsageError(f"unknown option {a}")
else:
opts["files"].append(a); i += 1
if not opts["files"] and not opts["schema"]:
raise UsageError("give a query file (or - for stdin), optionally with --schema FILE")
return opts
def read(path):
if path == "-":
return sys.stdin.read()
with open(path, encoding="utf-8") as f:
return f.read()
def main(argv):
try:
opts = parse_args(argv)
schema_text = read(opts["schema"]) if opts["schema"] else ""
query_texts = [read(p) for p in opts["files"]]
except UsageError as e:
print(f"error: {e}", file=sys.stderr)
return 2
except OSError as e:
print(f"error: cannot read {e.filename}: {e.strerror}", file=sys.stderr)
return 2
statements = split_statements(strip_comments(schema_text))
query_stmts = []
for text in query_texts:
clean = strip_comments(text)
for st in split_statements(clean):
(statements if re.match(r"\s*create\s", st, re.I) else query_stmts).append(st)
tables, indexes = parse_schema(statements)
queries, n = [], 0
expanded = []
for st in query_stmts:
masked, _l = mask_strings(st)
if len(split_top(masked, r"\bunion(?:\s+all)?\b")) > 1: # a real top-level UNION, not one inside a string
expanded.extend(split_top(st, r"\bunion(?:\s+all)?\b")) # each branch is analyzed as its own query
else:
expanded.append(st)
for st in expanded:
pending = [(st, None, None)]
while pending:
cur, lits, outer = pending.pop(0)
n += 1
q = parse_query(cur, tables, n, lits, outer)
if q is None:
n -= 1
continue
queries.append(q)
for sub in q["subqueries"]:
pending.append((sub, q["lits"], {**(outer or {}), **q["aliases"]}))
q["notes"].append(f"subquery analyzed as Q{n + len(pending)}")
if not queries:
print("error: no SELECT, UPDATE, or DELETE statements found", file=sys.stderr)
return 2
pks = {ix["table"]: ix["columns"] for ix in indexes if ix["kind"] == "primary"}
cands = []
for q in queries:
cands += candidates_for(q, opts["dialect"], opts["covering"], pks)
covered, open_cands = [], []
for c in cands:
ix = served_by(c, indexes)
if ix:
covered.append({**c, "existing": ix["name"]})
else:
open_cands.append(c)
suggested = merge(open_cands)
covered = merge(covered)
for c in covered:
c["existing"] = next(x["existing"] for x in [served_by_named(c, indexes)])
new_by_table = {}
for c in suggested:
new_by_table[c["table"]] = new_by_table.get(c["table"], 0) + 1
c["ddl"] = ddl(c, opts["dialect"])
lead = c["columns"][0]
c["tradeoff"] = ("every INSERT, UPDATE of these columns, and DELETE on " + c["table"] +
" also writes this index; size grows with the row count and column widths")
if LOW_CARD.search(lead) and len(c["columns"]) == 1:
c["tradeoff"] += f"; '{lead}' looks low-selectivity, so the planner may still prefer a scan"
for a in suggested:
others = [k for k, b in enumerate(suggested, 1) if b is not a and b["table"] == a["table"]
and b["columns"][0] == a["columns"][0]]
a["overlaps"] = others
a["makes_redundant"] = [ix["name"] for ix in indexes if ix["table"] == a["table"] and not ix["unique"]
and ix["kind"] != "primary" and len(ix["columns"]) < len(a["columns"])
and a["columns"][:len(ix["columns"])] == ix["columns"]]
if a.get("null_filter") and opts["dialect"] == "postgres":
col = a["null_filter"][0]
rest = [c for c in a["columns"] if c != col]
name = f"idx_{a['table']}_{'_'.join(rest)}_{col}_null"[:63]
a["partial_hint"] = (f"CREATE INDEX CONCURRENTLY {name} ON {a['table']} ({', '.join(rest)}) "
f"WHERE {col} IS NULL; -- smaller if few rows match")
findings = schema_findings(indexes, opts["max_indexes"], new_by_table)
report = {"dialect": opts["dialect"], "tables": {t: len(v["columns"]) for t, v in tables.items()},
"existing_indexes": indexes, "queries": [{k: v for k, v in q.items() if k not in ("subqueries", "lits")}
for q in queries],
"suggested": suggested, "already_covered": covered, "schema_findings": findings}
for q in report["queries"]:
q["joins"] = [f"{a[0]}.{a[1]} = {b[0]}.{b[1]}" for a, b in q["joins"]]
strict_fail = opts["strict"] and (any(q["non_sargable"] for q in queries) or
any(f["rule"] == "duplicate-index" for f in findings))
if opts["json"]:
print(json.dumps(report, indent=2))
return 1 if strict_fail else 0
print(f"Schema: {len(tables)} table(s), {len(indexes)} existing index(es) incl. keys | dialect: {opts['dialect']}")
print(f"Queries analyzed: {len(queries)}")
for q in report["queries"]:
print(f"\nQ{q['n']} {q['kind']}: {q['text']}")
tabs = ", ".join(f"{t} {a}" if a != t else t for a, t in q["aliases"].items())
print(f" tables: {tabs or '-'}")
eqs = [f"{f['table'] or '?'}.{f['column']} ({f['op']})" for f in q["filters"] if f["kind"] == "eq"]
rngs = [f"{f['table'] or '?'}.{f['column']} ({f['op']})" for f in q["filters"] if f["kind"] == "range"]
if eqs:
print(f" equality: {', '.join(eqs)}")
if rngs:
print(f" range: {', '.join(rngs)}")
if q["joins"]:
print(f" joins: {', '.join(q['joins'])}")
if q["order_by"]:
print(" order by: " + ", ".join(f"{o['table']}.{o['column']} {o['dir'].upper()}" for o in q["order_by"])
+ (" (LIMIT)" if q["limit"] else ""))
if q["group_by"]:
print(" group by: " + ", ".join(f"{g['table']}.{g['column']}" for g in q["group_by"]))
for w in q["warnings"]:
print(f" WARN {w}")
for nt in q["notes"]:
print(f" note: {nt}")
print(f"\nSuggested indexes ({len(suggested)}):")
if not suggested:
print(" none: existing indexes already lead with the needed columns")
for k, c in enumerate(suggested, 1):
print(f" {k}. {c['ddl']}")
print(f" serves: {', '.join('Q' + str(x) for x in c['queries'])} | why: {c['why']}")
if c["include"]:
print(f" covering: {', '.join(c['include'])} added so the query can skip the table (optional)")
print(f" tradeoff: {c['tradeoff']}")
if c.get("overlaps"):
print(f" overlap: same leading column as #{', #'.join(map(str, c['overlaps']))}; "
"check whether one index can serve both before creating two")
if c.get("makes_redundant"):
print(f" makes redundant: {', '.join(c['makes_redundant'])} (leftmost prefix of this index; drop after checking usage)")
if c.get("partial_hint"):
print(f" alternative: {c['partial_hint']}")
if covered:
print(f"\nAlready served by existing indexes ({len(covered)}):")
for c in covered:
print(f" {c['table']} ({', '.join(c['columns'])}) -> {c['existing']} (Q{', Q'.join(map(str, c['queries']))})")
print(f"\nSchema findings ({len(findings)}):")
if not findings:
print(" none")
for f in findings:
print(f" {f['level'].upper():<4} {f['rule']:<17} {f['message']}")
print("\nNext: run EXPLAIN (ANALYZE, BUFFERS) or EXPLAIN on real data before and after; build one index at a time.")
return 1 if strict_fail else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))