【中文说明】用于审查 Dockerfile 的安全性、镜像体积、构建缓存和运行可靠性，并提出修改版本及忽略文件建议。粘贴 Dockerfile，补充应用类型与构建环境，即可排查高权限运行、密钥泄露、缓存失效和依赖安装等问题。 Reviews Dockerfiles for security, image size, build cache, and runtime reliability: secrets baked into layers, running as root, unpinned base images, curl piped to a shell, ADD misuse, leftover apt and pip caches, cache-busting dependency installs, shell-form CMD, and missing multi-stage builds or health checks, then writes a corrected Dockerfile and .dockerignore. Use when a user shares a Dockerfile and asks "review my Dockerfile", "why is my image so big?", or "is this container secure?".
---
name: dockerfile-best-practices-reviewer
description: Reviews Dockerfiles for security, image size, build cache, and runtime reliability: secrets baked into layers, running as root, unpinned base images, curl piped to a shell, ADD misuse, apt and pip caches left in layers, dependency installs that bust the cache, shell-form CMD, and missing multi-stage builds or health checks, then writes a corrected Dockerfile and .dockerignore. Use when a user shares a Dockerfile and asks "review my Dockerfile", "why is my image so big?", "is this container secure?", or "why does every build reinstall dependencies?". Includes a tested stdlib Python linter.
---
# Dockerfile Best Practices Reviewer
You help developers ship container images that are small, fast to rebuild, and safe to run. You read a Dockerfile the way a careful platform engineer would: you find what leaks secrets or gives away root, what makes the image heavy, and what forces slow rebuilds, and you return a corrected Dockerfile that still builds and runs the same app.
## Files in this skill
- `scripts/lint_dockerfile.py` - parses a Dockerfile (continuation lines, multi-stage builds) and reports HIGH, WARN, and INFO findings with a fix for each (Python 3 standard library only)
- `references/dockerfile-rules.md` - every rule the linter checks, why it matters, and the correct pattern
- `references/language-patterns.md` - proven multi-stage patterns for Node.js, Python, Go, and Java, plus a starter .dockerignore
- `templates/dockerfile-review.md` - the review report to return
- `examples/example-node-api-review.md` - a worked review of a Node.js API Dockerfile
## Workflow
### 1. Collect the facts
Ask, or assume and say so:
- The Dockerfile, and the .dockerignore if there is one.
- Language and framework, how the app is built and started, and the port.
- Where it runs (Kubernetes, ECS, a VM with Docker Compose, a laptop) and whether that platform already has health checks.
- Constraints: required base images (company registry, distroless, Alpine or Debian), native dependencies, and whether the build uses BuildKit.
### 2. Run the linter
```bash
python3 scripts/lint_dockerfile.py Dockerfile
python3 scripts/lint_dockerfile.py services/*/Dockerfile --fail-on warn # stricter, for CI
python3 scripts/lint_dockerfile.py - --json < Dockerfile
```
The linter is a fast first pass based on patterns, not a full build. Read the whole Dockerfile yourself as well: it cannot know whether a package is really needed, whether a file copied in contains secrets, or whether the app writes to a path that a non-root user cannot access. If you cannot run the script, apply `references/dockerfile-rules.md` by hand.
### 3. Fix in this order
1. Security (HIGH): remove literal secrets from ENV, ARG, and RUN and use BuildKit secret mounts or runtime configuration instead; tell the user to rotate any secret that was ever committed or pushed in an image. Add a non-root USER. Replace curl-pipe-shell and chmod 777.
2. Reproducibility: pin base images to a version tag (digest for high-assurance builds) and use lockfile installs (npm ci, pip with a requirements lock, go mod download).
3. Build cache: copy dependency manifests first, install, then copy the source. Combine update and install in one RUN.
4. Size: multi-stage build, slim or distroless runtime base, no recommended packages, clean package caches in the same layer, a good .dockerignore.
5. Runtime: exec-form CMD or ENTRYPOINT, one process per container, a HEALTHCHECK if the platform has no probe, sensible WORKDIR and file ownership.
Rerun the linter on the corrected file until there are no HIGH findings and every WARN is fixed or explained.
### 4. Deliver
Fill in `templates/dockerfile-review.md`: summary, findings table, the corrected Dockerfile with short comments, a .dockerignore, how to build and test it, and an estimate of the size and rebuild-time impact. Use patterns from `references/language-patterns.md` and follow the style of `examples/example-node-api-review.md`.
## Rules
- Keep the app's behavior: same start command, port, environment variables, and files at runtime, unless the user agrees to change them.
- Never print or repeat a real secret found in the file; refer to it by name and tell the user to rotate it.
- Do not claim exact image sizes without a build; give ranges and say how to measure (`docker image ls`, `docker history`).
- Prefer official, maintained base images; mention the trade-offs of Alpine (musl) versus Debian slim versus distroless for the user's language.
- Explain each change in one line so the user learns the rule, not just the fix.
FILE:references/dockerfile-rules.md
# Dockerfile rules checked by the linter
Each rule: what is detected, why it matters, and the pattern to use instead.
## HIGH (security)
### secret-in-env / secret-in-run
Detected: ENV or ARG with a name like PASSWORD, SECRET, TOKEN, API_KEY, ACCESS_KEY, PRIVATE_KEY, CREDENTIALS and a literal value; `--password=...` or `user:pass@` URLs in RUN; ARG names that look like secrets (WARN).
Why: every ENV value and build arg is stored in the image config and history (`docker history --no-trunc`). Anyone who can pull the image can read it. Deleting it in a later layer does not remove it.
Use instead:
```dockerfile
# syntax=docker/dockerfile:1
RUN --mount=type=secret,id=npm_token \
NPM_TOKEN="$(cat /run/secrets/npm_token)" npm ci
```
Build with `docker build --secret id=npm_token,src=$HOME/.npm_token .`. Pass runtime secrets as environment variables from the platform or a secrets manager. Rotate any secret that was ever in an image.
### root-user
Detected: no USER in the final stage, or USER root / 0.
Why: a process that escapes the app as root inside the container has far more power over the host and other containers.
Use instead: create a user and switch before CMD; many official images already include one (`node`, `nobody`).
```dockerfile
RUN groupadd -r app && useradd -r -g app -u 10001 app
COPY --chown=app:app . /app
USER app
```
### curl-pipe-shell
Detected: `curl ... | sh` or `wget ... | bash`.
Why: runs whatever the server returns, with no integrity check; a compromised or changed script ends up in every build.
Use instead: download, verify a checksum, then run.
```dockerfile
RUN curl -fsSLo /tmp/install.sh https://example.com/install.sh \
&& echo "<sha256> /tmp/install.sh" | sha256sum -c - \
&& sh /tmp/install.sh && rm /tmp/install.sh
```
### chmod-777
Detected: `chmod 777` or `chmod -R 777`.
Why: any user or process in the container can modify the files, including app code.
Use instead: `COPY --chown=app:app` and `chmod 755` for executables, `644` for files.
## WARN
| Rule | Detected | Why | Fix |
|---|---|---|---|
| unpinned-base | FROM without tag or with :latest | builds change silently; hard to reproduce or roll back | `FROM node:20.11-bookworm-slim` or pin a digest `@sha256:...` |
| use-copy | ADD for local files or URLs | ADD auto-extracts archives and downloads without checksums | COPY for files; curl with checksum or `ADD --checksum=` for URLs; ADD is fine for local tar archives you want extracted |
| update-alone | `apt-get update` (or apk/yum) without install in the same RUN | the cached update layer gets reused with a newer install line, so packages come from a stale index | `RUN apt-get update && apt-get install -y ...` |
| cache-bust | dependency install after `COPY . .` | any source change invalidates the install layer, so every build reinstalls everything | copy manifests first, install, then copy the source |
| sudo | sudo in RUN | not needed (build runs as root until USER), adds a setuid binary | run the step before USER |
| ssh-port | EXPOSE 22 | containers should not run SSH daemons | `docker exec`, `kubectl exec` |
| multiple-cmd | more than one CMD or ENTRYPOINT in a stage | only the last one counts; usually a mistake | keep one |
## INFO
| Rule | Fix |
|---|---|
| apt-recommends | `apt-get install -y --no-install-recommends ...` |
| apt-lists | end the same RUN with `&& rm -rf /var/lib/apt/lists/*` |
| apk-cache | `apk add --no-cache ...` |
| pip-cache | `pip install --no-cache-dir ...` or a BuildKit cache mount |
| npm-ci | `npm ci --omit=dev` uses the lockfile exactly |
| shell-form | `CMD ["node", "server.js"]` so the app is PID 1 and receives SIGTERM for graceful shutdown |
| cd-in-run | `WORKDIR /app` |
| no-healthcheck | `HEALTHCHECK CMD curl -fsS http://localhost:3000/health || exit 1` if the platform has no probe; Kubernetes ignores HEALTHCHECK and uses its own probes |
| single-stage | build in a builder stage; copy only the output into a slim runtime stage |
| many-layers | combine related RUN steps; fewer, purposeful layers |
| dockerignore | exclude `.git`, `node_modules`, `.env*`, build output, logs, and local secrets |
## What the linter cannot see
- Whether copied files contain secrets (check .dockerignore and the repo).
- Vulnerable packages in the base image or dependencies: suggest an image scanner in CI.
- Whether the non-root user can write where the app writes (logs, uploads, caches).
- Architecture issues (arm64 vs amd64), native modules, and Alpine musl compatibility.
FILE:references/language-patterns.md
# Multi-stage patterns by language
Adapt versions to the project. All examples run as a non-root user, use exec-form CMD, and keep build tools out of the runtime image.
## Node.js (npm)
```dockerfile
# syntax=docker/dockerfile:1
FROM node:20.11-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
FROM node:20.11-bookworm-slim AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
FROM node:20.11-bookworm-slim
ENV NODE_ENV=production
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build --chown=node:node /app/dist ./dist
USER node
EXPOSE 3000
CMD ["node", "dist/server.js"]
```
Skip the build stage if there is no compile step. Use `node:<version>-alpine` only if native modules work with musl.
## Python (pip)
```dockerfile
FROM python:3.12-slim AS build
WORKDIR /app
RUN python -m venv /venv
ENV PATH="/venv/bin:$PATH"
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
FROM python:3.12-slim
ENV PATH="/venv/bin:$PATH" PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
WORKDIR /app
COPY --from=build /venv /venv
RUN useradd -r -u 10001 app
COPY --chown=app:app . .
USER app
EXPOSE 8000
CMD ["gunicorn", "-b", "0.0.0.0:8000", "app:app"]
```
If packages need compilers (psycopg2, numpy from source), install `build-essential` only in the build stage.
## Go
```dockerfile
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/app
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
USER nonroot
ENTRYPOINT ["/app"]
```
A static Go binary on distroless or scratch gives images of a few MB to tens of MB.
## Java (Maven)
```dockerfile
FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /src
COPY pom.xml .
RUN mvn -q dependency:go-offline
COPY src ./src
RUN mvn -q package -DskipTests
FROM eclipse-temurin:21-jre
RUN useradd -r -u 10001 app
WORKDIR /app
COPY --from=build --chown=app:app /src/target/app.jar app.jar
USER app
EXPOSE 8080
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75", "-jar", "app.jar"]
```
## Starter .dockerignore
```
.git
.gitignore
.dockerignore
Dockerfile*
node_modules
dist
build
target
__pycache__
*.pyc
.venv
.env
.env.*
*.log
coverage
.idea
.vscode
secrets/
*.pem
*.key
```
## Measuring the result
- `docker build -t app:review .` then `docker image ls app:review` for size.
- `docker history app:review` to see which layers are large.
- Change one source file and rebuild: the dependency install step should say CACHED.
FILE:templates/dockerfile-review.md
# Dockerfile review: service_name
**App:** language_and_framework, started with start_command, port port
**Runs on:** platform
**Linter:** HIGH high / WARN warn / INFO info before; HIGH 0 / WARN warn_after / INFO info_after after
## Summary
Two or three sentences: the biggest security problem, the biggest size or speed problem, and the expected improvement.
## Findings
| # | Level | Line | Problem | Fix |
|---|---|---|---|---|
| 1 | HIGH | | | |
## Action needed outside the Dockerfile
- Rotate: (names of any secrets that were in the file or image, never their values)
- Platform: health probes, read-only filesystem, resource limits
## Corrected Dockerfile
```dockerfile
(full corrected Dockerfile with one-line comments on changed parts)
```
## .dockerignore
```
(entries)
```
## Build and verify
```bash
docker build -t image:review .
docker run --rm -p port:port image:review
docker image ls image:review
python3 scripts/lint_dockerfile.py Dockerfile
```
## Expected impact (estimate)
- Image size: from about ... to about ...
- Rebuild after a source change: dependency layer cached
- Security: non-root, no secrets in layers, pinned base
FILE:examples/example-node-api-review.md
# Example: review of a Node.js API Dockerfile
## User request
"Our orders API image is 1.2 GB and every build reinstalls all npm packages, even when I change one line. Can you review the Dockerfile? It runs on Kubernetes, Express app, `node server.js`, port 3000. No native modules as far as I know."
## Input Dockerfile
```dockerfile
FROM node
ENV NODE_ENV=production
ENV DB_PASSWORD=supersecret123
RUN apt-get update
RUN apt-get install -y curl git build-essential
WORKDIR /app
COPY . .
RUN npm install
RUN curl -fsSL https://example.com/install-agent.sh | bash
RUN chmod -R 777 /app
ADD config.json /app/config.json
EXPOSE 3000 22
CMD node server.js
```
## Linter run
```
$ python3 scripts/lint_dockerfile.py Dockerfile
Dockerfile: 1 stage(s): node
HIGH 4 WARN 5 INFO 7
HIGH line 1 root-user the final image runs as root
HIGH line 3 secret-in-env ENV DB_PASSWORD contains a literal secret; it is stored in the image layers and history
HIGH line 9 curl-pipe-shell downloads a script and pipes it straight into a shell
HIGH line 10 chmod-777 chmod 777 makes files writable by every user
WARN line 1 unpinned-base base image 'node' has no tag, so it means :latest and changes without notice
WARN line 4 update-alone 'apt-get update' runs in its own layer; a later install may use a stale cached index
WARN line 8 cache-bust dependencies are installed after 'COPY . .' (line 7), so any source change reinstalls them
WARN line 11 use-copy ADD is used for local files; it also auto-extracts archives and fetches URLs
WARN line 12 ssh-port port 22 (SSH) is exposed; containers should not run an SSH server
INFO line 1 no-healthcheck no HEALTHCHECK in the final stage
INFO line 1 single-stage build tools are installed in the only stage, so they ship in the final image
INFO line 5 apt-recommends apt installs recommended packages too
INFO line 5 apt-lists apt package lists are left in the layer (tens of MB)
INFO line 7 dockerignore COPY of the whole build context
INFO line 8 npm-ci npm install can change the lockfile and versions
INFO line 13 shell-form shell-form CMD runs under /bin/sh -c, so the app is not PID 1 and may miss SIGTERM
```
## Review (filled template, shortened)
**Linter:** HIGH 4 / WARN 5 / INFO 7 before; HIGH 0 / WARN 0 / INFO 0 after
**Summary:** The image ships a database password in its layers and runs as root, and it pipes a remote script into bash. It is large because it uses the full `node` image with compilers and git, and slow because `COPY . .` comes before `npm install`. The corrected two-stage Dockerfile removes the secret, runs as the built-in `node` user, and caches dependencies; expect roughly 200 to 300 MB instead of 1.2 GB (measure with `docker image ls`).
| # | Level | Line | Problem | Fix |
|---|---|---|---|---|
| 1 | HIGH | 3 | DB_PASSWORD literal in ENV, visible in `docker history` | removed; inject from a Kubernetes Secret at runtime |
| 2 | HIGH | - | runs as root | `USER node` (built into the official image) |
| 3 | HIGH | 9 | remote install script piped into bash | removed: the agent now runs as a Kubernetes sidecar; if it must stay, download, verify the checksum, then run |
| 4 | HIGH | 10 | chmod -R 777 /app | `COPY --chown=node:node` instead |
| 5 | WARN | 1 | untagged `node` base | `node:20.11-bookworm-slim` |
| 6 | WARN | 4 | apt-get update in its own layer | combined with install and list cleanup |
| 7 | WARN | 8 | npm install after `COPY . .` | manifests first, `npm ci --omit=dev` in a deps stage |
| 8 | WARN | 11 | ADD for a local file | COPY |
| 9 | WARN | 12 | EXPOSE 22 | removed; use `kubectl exec` |
| 10 | INFO | 5, 13 | build tools and git in the runtime image, shell-form CMD | dropped (no native modules), exec-form CMD |
**Action needed outside the Dockerfile:** rotate DB_PASSWORD now (it is in every image pushed so far), and add it to the Deployment from a Kubernetes Secret. Kubernetes ignores HEALTHCHECK, so also add a readiness and liveness probe on /health; the HEALTHCHECK below helps local Docker and Compose.
## Corrected Dockerfile
```dockerfile
# syntax=docker/dockerfile:1
FROM node:20.11-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
FROM node:20.11-bookworm-slim
ENV NODE_ENV=production
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl \
&& rm -rf /var/lib/apt/lists/*
COPY --from=deps /app/node_modules ./node_modules
COPY --chown=node:node src/ ./src/
COPY --chown=node:node config.json ./
USER node
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=3s CMD curl -fsS http://localhost:3000/health || exit 1
CMD ["node", "src/server.js"]
```
## .dockerignore
```
.git
node_modules
.env
.env.*
*.log
coverage
Dockerfile*
```
## Verify
```
$ python3 scripts/lint_dockerfile.py Dockerfile
Dockerfile: 2 stage(s): node:20.11-bookworm-slim AS deps, node:20.11-bookworm-slim
HIGH 0 WARN 0 INFO 0
no problems found
```
Note: the source now lives in `src/`, so the start command became `node src/server.js`; if your entry file is elsewhere, keep your original path. If a dependency turns out to need compilers, install `build-essential` and `python3` in the deps stage only.
FILE:scripts/lint_dockerfile.py
#!/usr/bin/env python3
"""Review a Dockerfile for security, image size, build cache, and reliability problems.
Usage:
python3 lint_dockerfile.py Dockerfile [more Dockerfiles] [--json] [--fail-on high|warn|info]
python3 lint_dockerfile.py - < Dockerfile
Checks (rule ids in brackets):
HIGH secrets in ENV/ARG or in RUN (secret-in-env, secret-in-run), final stage runs as root
(root-user), pipe from curl/wget to a shell (curl-pipe-shell), chmod 777 (chmod-777)
WARN untagged or :latest base image (unpinned-base), ADD for local files or URLs (use-copy),
apt/apk/yum update without install in the same RUN (update-alone), COPY of the whole
context before installing dependencies (cache-bust), sudo (sudo), EXPOSE 22 (ssh-port),
more than one CMD or ENTRYPOINT in a stage (multiple-cmd)
INFO apt-get without --no-install-recommends or without cleaning lists, pip without
--no-cache-dir, npm install instead of npm ci, shell-form CMD/ENTRYPOINT, cd in RUN
instead of WORKDIR, no HEALTHCHECK, single-stage build with build tools, many RUN layers,
COPY of the whole context (check .dockerignore)
Exit codes: 0 ok, 1 findings at or above --fail-on (default high), 2 usage or read error.
Python 3 standard library only.
"""
import json
import re
import shlex
import sys
LEVELS = {"info": 0, "warn": 1, "high": 2}
SECRET_KEY = re.compile(r"(PASSWORD|PASSWD|SECRET|TOKEN|API_?KEY|ACCESS_?KEY|PRIVATE_?KEY|CREDENTIALS?)", re.I)
BUILD_TOOLS = re.compile(r"\b(build-essential|gcc|g\+\+|make|cmake|maven|gradle|golang|rustc|cargo|npm run build|go build|mvn )", re.I)
def logical_lines(text):
"""Join continuation lines; return list of (line_no, instruction, args). Skips comments and parser directives."""
out, buf, start = [], "", None
escape = "\\"
for i, raw in enumerate(text.replace("\r\n", "\n").split("\n"), 1):
m = re.match(r"^#\s*escape\s*=\s*(\S)", raw)
if m and not out and not buf:
escape = m.group(1)
continue
s = raw.strip()
if not buf and (not s or s.startswith("#")):
continue
if buf and s.startswith("#"):
continue
if start is None:
start = i
if s.endswith(escape):
buf += s[:-1] + " "
continue
buf += s
parts = buf.split(None, 1)
out.append((start, parts[0].upper(), parts[1] if len(parts) > 1 else ""))
buf, start = "", None
if buf:
parts = buf.split(None, 1)
out.append((start, parts[0].upper(), parts[1] if len(parts) > 1 else ""))
return out
def review(text):
lines = logical_lines(text)
findings = []
def add(level, rule, line, msg, fix):
findings.append({"level": level, "rule": rule, "line": line, "message": msg, "fix": fix})
if not any(ins == "FROM" for _, ins, _ in lines):
raise ValueError("no FROM instruction found (is this a Dockerfile?)")
stages, cur = [], None
for ln, ins, args in lines:
if ins == "FROM":
m = re.match(r"(?:--platform=\S+\s+)?(\S+)(?:\s+AS\s+(\S+))?", args, re.I)
cur = {"line": ln, "image": m.group(1) if m else args, "name": m.group(2) if m else None, "ins": []}
stages.append(cur)
elif cur is not None:
cur["ins"].append((ln, ins, args))
names = {s["name"].lower() for s in stages if s["name"]}
arg_names = {a.split("=")[0].strip() for _, i, a in lines if i == "ARG"}
for s in stages:
img = s["image"]
if img.lower() in names or img == "scratch" or img.startswith("$"):
pass
elif "@sha256:" in img:
pass
elif ":" not in img.split("/")[-1]:
add("warn", "unpinned-base", s["line"], f"base image '{img}' has no tag, so it means :latest and changes without notice",
f"pin a version, e.g. {img}:<version>-slim, or a digest")
elif img.endswith(":latest"):
add("warn", "unpinned-base", s["line"], f"base image '{img}' uses :latest", "pin a specific version tag or digest")
final = stages[-1]
run_count = 0
for idx, s in enumerate(stages):
cmds = [x for x in s["ins"] if x[1] in ("CMD", "ENTRYPOINT")]
for kind in ("CMD", "ENTRYPOINT"):
k = [x for x in cmds if x[1] == kind]
if len(k) > 1:
add("warn", "multiple-cmd", k[-1][0], f"{len(k)} {kind} instructions in one stage; only the last one counts",
f"keep a single {kind}")
copied_all = None
for ln, ins, args in s["ins"]:
low = args.lower()
if ins in ("ENV", "ARG"):
pairs = re.findall(r"([A-Za-z_][A-Za-z0-9_]*)(?:(?:=|\s+)(\"[^\"]*\"|'[^']*'|[^\s=]\S*))?", args) if ins == "ARG" or "=" in args else [tuple((args.split(None, 1) + [""])[:2])]
for key, val in pairs:
if SECRET_KEY.search(key) and val and not val.startswith("$") and val.strip("\"'"):
add("high", "secret-in-env", ln, f"{ins} {key} contains a literal secret; it is stored in the image layers and history",
"pass secrets at build time with RUN --mount=type=secret, or at run time via environment or a secrets manager")
if ins == "ARG" and any(SECRET_KEY.search(k) for k, _ in pairs) and not any(v for k, v in pairs if SECRET_KEY.search(k)):
add("warn", "secret-in-env", ln, f"ARG {pairs[0][0]} looks like a secret; build args are visible in image history",
"use RUN --mount=type=secret instead of a build arg")
if ins == "RUN":
run_count += 1
if re.search(r"(curl|wget)\b[^|;&]*\|\s*(sudo\s+)?(ba|z|da)?sh\b", low):
add("high", "curl-pipe-shell", ln, "downloads a script and pipes it straight into a shell",
"download to a file, verify a checksum or signature, then run it")
if re.search(r"chmod\s+(-r\s+)?0?777\b", low):
add("high", "chmod-777", ln, "chmod 777 makes files writable by every user", "grant only the permissions needed, e.g. chmod 755 or chown to the app user")
if re.search(r"(--password[= ]\S+|://[^/\s:@]+:[^@\s$]+@)", args) or re.search(r"\b(token|api[_-]?key)=\w{8,}", low):
add("high", "secret-in-run", ln, "a password, token, or credentials URL appears in a RUN command and stays in the image history",
"use RUN --mount=type=secret or fetch credentials at run time")
if re.search(r"\bsudo\b", low):
add("warn", "sudo", ln, "sudo inside a Dockerfile is unnecessary and widens the attack surface", "run the step before switching USER, without sudo")
for upd, inst in (("apt-get update", "apt-get install"), ("apt update", "apt install"), ("apk update", "apk add"), ("yum update", "yum install")):
if upd in low and inst not in low:
add("warn", "update-alone", ln, f"'{upd}' runs in its own layer; a later install may use a stale cached index",
f"combine: RUN {upd} && {inst} ... in one RUN")
if "apt-get install" in low or "apt install" in low:
if "--no-install-recommends" not in low:
add("info", "apt-recommends", ln, "apt installs recommended packages too", "add --no-install-recommends")
if "/var/lib/apt/lists" not in low:
add("info", "apt-lists", ln, "apt package lists are left in the layer (tens of MB)", "end the same RUN with && rm -rf /var/lib/apt/lists/*")
if "apk add" in low and "--no-cache" not in low:
add("info", "apk-cache", ln, "apk cache is kept in the layer", "use apk add --no-cache")
if re.search(r"\bpip3?\s+install\b", low) and "--no-cache-dir" not in low and "--mount=type=cache" not in low:
add("info", "pip-cache", ln, "pip keeps its download cache in the layer", "add --no-cache-dir (or use a cache mount)")
if any(re.fullmatch(r"\s*npm\s+(install|i)(\s+--?(?!g\b|global\b)[\w-]+(=\S+)?)*\s*", seg) for seg in re.split(r"&&|;|\|\|", low)):
add("info", "npm-ci", ln, "npm install can change the lockfile and versions", "use npm ci (with --omit=dev for production)")
if re.match(r"cd\s+\S+\s*&&", low) or re.search(r"&&\s*cd\s+/", low):
add("info", "cd-in-run", ln, "cd in RUN only affects that one command", "use WORKDIR /path")
if copied_all and re.search(r"\b(npm (ci|install)|pip3? install -r|poetry install|bundle install|go mod download|composer install|yarn install)\b", low):
add("warn", "cache-bust", ln, f"dependencies are installed after 'COPY {copied_all[1]}' (line {copied_all[0]}), so any source change reinstalls them",
"copy only the manifest and lockfile first (e.g. COPY package*.json ./), install, then COPY the rest")
copied_all = None
if ins == "ADD":
src = args.split()[0] if args.split() else ""
if re.match(r"https?://", src):
add("warn", "use-copy", ln, "ADD with a URL downloads without checksum verification and leaves the file in a layer",
"use RUN curl -fsSL ... with a checksum check, or ADD --checksum=sha256:...")
elif not re.search(r"\.(tar|tar\.gz|tgz|tar\.xz|tar\.bz2)$", src):
add("warn", "use-copy", ln, "ADD is used for local files; it also auto-extracts archives and fetches URLs", "use COPY for plain files")
if ins == "COPY" and not args.startswith("--from"):
parts = [p for p in args.split() if not p.startswith("--")]
if parts and parts[0] in (".", "./", "*"):
copied_all = (ln, " ".join(parts))
add("info", "dockerignore", ln, "COPY of the whole build context", "make sure .dockerignore excludes .git, node_modules, .env, build output, and secrets")
if ins == "EXPOSE" and re.search(r"\b22\b", args):
add("warn", "ssh-port", ln, "port 22 (SSH) is exposed; containers should not run an SSH server", "use docker exec / kubectl exec for access")
if ins in ("CMD", "ENTRYPOINT") and not args.strip().startswith("["):
add("info", "shell-form", ln, f"shell-form {ins} runs under /bin/sh -c, so the app is not PID 1 and may miss SIGTERM",
f'use exec form: {ins} ["executable", "arg"]')
users = [(ln, a.strip()) for ln, i, a in final["ins"] if i == "USER"]
if not users or users[-1][1].split(":")[0] in ("root", "0"):
where = users[-1][0] if users else final["line"]
add("high", "root-user", where, "the final image runs as root",
"create an unprivileged user (e.g. RUN useradd -r -u 10001 app) and add USER app before CMD")
if not any(i == "HEALTHCHECK" for _, i, _ in final["ins"]):
add("info", "no-healthcheck", final["line"], "no HEALTHCHECK in the final stage", "add HEALTHCHECK CMD ... if no orchestrator health probe is configured")
if len(stages) == 1 and any(BUILD_TOOLS.search(a) for _, i, a in final["ins"] if i == "RUN"):
add("info", "single-stage", final["line"], "build tools are installed in the only stage, so they ship in the final image",
"use a multi-stage build: compile in a builder stage, COPY --from=builder only the output")
if run_count > 10:
add("info", "many-layers", final["line"], f"{run_count} RUN instructions; related steps can be combined", "group related commands with && in fewer RUN steps")
return {"stages": [{"line": s["line"], "image": s["image"], "name": s["name"]} for s in stages], "findings": findings}
def main(argv):
as_json, fail_on, files = False, "high", []
it = iter(argv)
for a in it:
if a == "--json":
as_json = True
elif a == "--fail-on":
fail_on = next(it, "").lower()
if fail_on not in LEVELS:
print("error: --fail-on must be high, warn, or info", file=sys.stderr)
return 2
elif a in ("-h", "--help"):
print(__doc__)
return 0
elif a.startswith("--"):
print(f"error: unknown option {a}", file=sys.stderr)
return 2
else:
files.append(a)
if not files:
print("error: give a Dockerfile path, or - for stdin", file=sys.stderr)
return 2
reports, worst = [], -1
for f in files:
try:
text = sys.stdin.read() if f == "-" else open(f, encoding="utf-8").read()
r = review(text)
except (OSError, UnicodeDecodeError, ValueError) as e:
print(f"error: {f}: {e}", file=sys.stderr)
return 2
r["file"] = f
r["findings"].sort(key=lambda x: (-LEVELS[x["level"]], x["line"]))
for x in r["findings"]:
worst = max(worst, LEVELS[x["level"]])
reports.append(r)
if as_json:
print(json.dumps(reports, indent=2))
else:
for r in reports:
c = {lv: sum(1 for x in r["findings"] if x["level"] == lv) for lv in LEVELS}
st = ", ".join(f"{s['image']}" + (f" AS {s['name']}" if s["name"] else "") for s in r["stages"])
print(f"{r['file']}: {len(r['stages'])} stage(s): {st}")
print(f" HIGH {c['high']} WARN {c['warn']} INFO {c['info']}")
for x in r["findings"]:
print(f" {x['level'].upper():<4} line {x['line']:<3} {x['rule']:<15} {x['message']}")
print(f" fix: {x['fix']}")
if not r["findings"]:
print(" no problems found")
return 1 if worst >= LEVELS[fail_on] else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))【中文说明】用于逐 token 解释 Python 正则表达式，测试必须匹配和不得匹配的样例，并检查灾难性回溯、未转义点号、空匹配、缺少锚点等风险。使用时提供正则、目标语言、正反例和期望捕获组，模型会展示匹配跨度与命名组，并在必要时给出更安全的改写。 Explains Python regular expressions token by token, tests them on must-match and must-not-match samples with spans and named groups, and flags risky patterns: nested quantifiers that backtrack catastrophically, [A-z], unescaped dots, empty matches, and missing anchors, then proposes a safer rewrite. Use when a user asks "what does this regex do?", "why doesn't it match?", or "is this regex safe?". Includes a tested stdlib Python checker.
--- name: regex-pattern-explainer-and-tester description: Explains Python regular expressions token by token, tests them on must-match and must-not-match samples with spans and named groups, and flags risky patterns: nested quantifiers that backtrack catastrophically, [A-z], unescaped dots, empty matches, and missing anchors, then proposes a safer rewrite. Use when a user asks "what does this regex do?", "why doesn't it match?", or "is this regex safe?". Includes a tested stdlib Python checker. --- # Regex Pattern Explainer and Tester You help people understand, fix, and trust regular expressions. You explain a pattern in plain words, prove what it does with real test cases, and find the traps that make regexes fail in production: inputs that should not pass, inputs that should pass but do not, and patterns that hang on long input. You return a corrected pattern with a test table, not just an opinion. ## Files in this skill - `scripts/regex_check.py` - explains a pattern from Python's own regex parser, runs sample inputs with "+" (must match) and "-" (must not match) expectations, and reports HIGH, WARN, and INFO risks; `--probe` times the pattern on generated worst-case input in a subprocess (Python 3 standard library only) - `references/regex-pitfalls.md` - every risk the checker reports, why it matters, and the safe pattern to use instead - `references/flavor-differences.md` - how Python `re` differs from JavaScript, PCRE, Go RE2, .NET, Java, and POSIX grep, so a fix works where the user runs it - `templates/regex-review.md` - the review report to return - `examples/example-email-validation-review.md` - a worked review of an email check that hangs on long input ## Workflow ### 1. Collect the facts Ask, or assume and say so: - The pattern exactly as written in the code (raw string or escaped string literal) and the language or tool that runs it. - What it must accept and what it must reject: 4 to 10 real examples of each, including edge cases (empty input, spaces, Unicode, very long input). - How it is used: validate a whole value (fullmatch), find the first match (search), extract all matches (findall), or replace. - Where the input comes from. Untrusted input (web forms, logs, API payloads) makes catastrophic backtracking a security issue. ### 2. Run the checker ```bash python3 scripts/regex_check.py 'PATTERN' -t 'sample one' -t 'sample two' python3 scripts/regex_check.py 'PATTERN' --cases cases.txt --mode fullmatch --flags a python3 scripts/regex_check.py 'PATTERN' --probe --no-explain # confirm a backtracking risk python3 scripts/regex_check.py 'PATTERN' --cases - --json < cases.txt ``` A cases file has one sample per line: `+ text` must match, `- text` must not match, `# ...` is a comment. Always quote the pattern in single quotes in the shell. The checker uses Python's `re`; for other flavors, read `references/flavor-differences.md` and translate the result. If you cannot run the script, explain the pattern by hand with the same structure and apply `references/regex-pitfalls.md`. ### 3. Explain, then fix 1. Explain the pattern top to bottom in plain words, one line per token or group, using the checker's explanation as the base. 2. Show the test table: each sample, expected result, actual result, and the matched text or groups. 3. Fix in this order: patterns that can hang (HIGH), wrong accepts and wrong rejects (failed expectations), then clarity and the INFO notes. 4. Prefer simple, readable rewrites: one repeat instead of nested ones, negated classes like `[^,]*` instead of `.*`, explicit anchors `\A...\Z` or `fullmatch`, and `re.VERBOSE` with comments for long patterns. 5. Rerun the checker on the corrected pattern with the same cases plus any new edge cases until every expectation passes and there is no HIGH risk. Run `--probe` on the fix if the original had a HIGH risk. ### 4. Deliver Fill in `templates/regex-review.md`: the plain-language explanation, the test table before and after, the risks found, the corrected pattern in the user's language with correct escaping, and a short list of test cases to keep in their test suite. Follow the style of `examples/example-email-validation-review.md`. ## Rules - Never claim a pattern is correct without showing it on test cases; add the edge cases the user forgot. - Do not over-validate: for emails, URLs, and names, recommend a simple shape check plus a real verification step (sending an email, parsing the URL) instead of a giant pattern. - Keep the user's flavor: do not suggest lookbehind, possessive quantifiers, or atomic groups where their engine does not support them. - Escape correctly for the target language (raw strings in Python, double backslashes in Java and C# strings, slashes in JavaScript literals). - Treat catastrophic backtracking on untrusted input as a security bug (ReDoS) and say so plainly; suggest input length limits as a second defense. FILE:references/regex-pitfalls.md # Regex pitfalls checked by the script Each rule: what is detected, why it matters, and the pattern to use instead. Levels: HIGH can hang or be exploited, WARN usually accepts or rejects the wrong input, INFO is a habit worth fixing. ## HIGH (catastrophic backtracking, ReDoS) ### nested-quantifier Detected: a repeat inside another repeat where nothing else in the outer repeat is required, such as `(a+)+`, `(\w+\s?)+`, `([a-z0-9]+\.?)+`, `(\w+|\d+)+`. Why: on input that almost matches (a long run of letters followed by a char the pattern rejects), the engine tries every way to split the run between the inner and outer repeat: 2^n ways for n chars. 30 chars can take minutes. On a web form or API this is a denial-of-service bug. Use instead: one repeat that describes the whole run, with the separator required. ``` (\w+\s?)+$ -> \w+(?:\s\w+)*$ ([a-z0-9]+\.?)+@ -> [a-z0-9]+(?:\.[a-z0-9]+)*@ ``` Python 3.11+ also supports atomic groups `(?>...)` and possessive quantifiers `a++`, which never give back what they matched. ### ambiguous-repeat Detected: a repeat whose body has variable length and whose required parts can also be eaten by the variable part, such as `(x+x+)+`, `(aa|a)+`, `(foo|f\w+)+`. Why: same as above, the iterations can be cut in many ways. Growth may be slower (Fibonacci instead of 2^n) but is still exponential. Use instead: make each iteration start with a char nothing else in the body can match, or merge the parts into one class. Confirm both with `--probe`: the script runs the pattern on `c * n + tail` for n = 12 to 28 in a subprocess with a 2 second limit and reports how fast the time grows. A growth of more than 3x per 4 extra chars, or a timeout, confirms the risk. ## WARN ### nested-overlap A repeat whose body has a separator that the inner part also accepts, such as `([\w.]+\.)+`. Make the inner class exclude the separator: `(?:[\w-]+\.)+`. ### adjacent-overlap Two unbounded single-char repeats in a row that accept the same chars, such as `\d+\d+` or `.*.*`. The split point is ambiguous and a failing match backtracks polynomially (n^2, n^3). Keep one of them or make the boundary explicit. ### range-A-z `[A-z]` covers ASCII 65 to 122, which includes `[ \ ] ^ _` and the backtick. Use `[A-Za-z]`, or `[^\W\d_]` for any Unicode letter. ### unescaped-dot `.` between letters or before a word such as `com` or `json` matches any char, so `example-com` passes a check for `example.com`. Write `\.`. ### matches-empty The pattern accepts the empty string. A validation then accepts blank input, and `findall` returns empty strings between real matches. Use `+` instead of `*` where at least one char is required. ## INFO ### not-anchored `re.search(r"\d+", "abc123xyz")` succeeds. For validating a whole value, use `re.fullmatch` or anchor with `\A ... \Z`. ### dollar-newline `$` matches at the end of the string and also just before a final newline, so `^abc$` accepts `"abc\n"` with `re.search` or `re.match`. Use `\Z` or `re.fullmatch` for strict checks. ### unicode-classes In Python 3, `\d` matches any Unicode digit (for example Arabic-Indic digits) and `\w` matches letters in every script. Good for names and free text; for IDs, card numbers, and codes use `re.ASCII` or `[0-9]`. ### greedy-dotstar and many-dotstar `.*` takes everything and then gives back one char at a time. Three or more `.*` in one pattern make failures slow on long lines. Describe each field with a negated class: `"[^"]*"` for a quoted string, `[^,]*` for a CSV field. ## Not detected by the script (check by hand) - Wrong escaping in the host language (a Python string without `r""`, a Java string with single backslashes). - Business rules a regex cannot express well: valid dates (February 30), checksums (IBAN, card numbers), real email deliverability. - Locale issues: case-insensitive matching of Turkish dotted and dotless i, German sharp s. - Performance of huge alternations (thousands of words): use a set lookup or a trie instead. FILE:references/flavor-differences.md # Regex flavor differences that break fixes The checker runs Python `re`. Before handing a pattern to another engine, check this table. | Feature | Python `re` | JavaScript | PCRE / PHP | Go (RE2) | .NET | Java | POSIX grep -E | |---|---|---|---|---|---|---|---| | Lookahead `(?=...)` | yes | yes | yes | no | yes | yes | no | | Lookbehind `(?<=...)` | fixed width only | yes (ES2018) | fixed or bounded | no | any width | bounded | no | | Named group | `(?P<name>...)` only | `(?<name>...)` | both | `(?P<name>...)` (also `(?<name>...)` in Go 1.22+) | `(?<name>...)` | `(?<name>...)` | no | | Atomic group `(?>...)` | 3.11+ | no | yes | no | yes | yes | no | | Possessive `a++` | 3.11+ | no | yes | no | no | yes | no | | Backreference `\1` | yes | yes | yes | no | yes | yes | yes (basic) | | `\d` matches non-ASCII digits | yes (unless re.ASCII) | no | no (unless UCP) | no | yes | no (unless UNICODE_CHARACTER_CLASS) | locale | | `$` before final newline | yes | no (without m flag) | yes | no | yes | yes | n/a (line based) | | Inline flags `(?i)` | yes, at start or scoped `(?i:...)` | flags after the literal (`/i`); scoped `(?i:...)` only in the newest engines | yes | yes | yes | yes | no | | Catastrophic backtracking | possible | possible | possible (has limits) | impossible (linear time) | possible (timeout option) | possible | engine dependent | | Full match helper | `re.fullmatch` | anchor with `^...$` | anchor | anchor | anchor | `String.matches()` | `-x` flag | ## Translation notes - JavaScript: write the literal as `/pattern/flags`; escape `/` inside it. Use the `u` flag for Unicode and `v` for set operations in newer engines. `\Z` does not exist: use `$` without the `m` flag. - Go RE2: no lookarounds or backreferences, but every pattern runs in linear time, so ReDoS is not a concern there. Rewrite lookarounds as a second check in code. - Java and C#: every backslash doubles in a normal string literal (`"\\d+"`); C# verbatim strings `@"\d+"` and Java text blocks help. - .NET: set `RegexOptions.NonBacktracking` (.NET 7+) or a match timeout for untrusted input. - SQL: PostgreSQL `~` uses Tcl-style advanced regexes (lookahead, lookbehind since 9.6, no named groups); MySQL 8 uses ICU. Test in the database itself. - Shell: `grep -E` has no `\d` on many systems; use `[0-9]`. `grep -P` gives PCRE where available. ## Safe defaults across flavors - `[0-9]` instead of `\d`, `[A-Za-z]` instead of `\w` when the format is ASCII by definition. - Anchors `^` and `$` without multiline mode, plus a separate length check (for example at most 254 chars for an email). - No nested quantifiers on untrusted input, whatever the engine. FILE:templates/regex-review.md # Regex review: purpose **Pattern (original):** `original_pattern` **Engine:** language_and_engine | **Used as:** validate whole value / find first / find all / replace **Checker:** HIGH high / WARN warn / INFO info before; expectations pass_before/total pass ## What it does, in plain words 1. (one line per token or group, top to bottom) ## Test results | # | Input | Expected | Before | After | Notes | |---|---|---|---|---|---| | 1 | | match / no match | | | | ## Risks found | Level | Rule | What it means here | Fix | |---|---|---|---| | HIGH | | | | Probe (if run): input `probe_input`, probe_result ## Corrected pattern ```language (the corrected pattern, escaped for the user's language, with a VERBOSE or commented version if it is longer than about 40 chars) ``` Why each change: (one line per change) ## Keep these tests ``` + (must match) - (must not match) ``` ## Notes - Limits of a regex for this job and the extra check to do in code (length limit, parsing, verification step). - Flavor caveats if the pattern will run outside Python. FILE:examples/example-email-validation-review.md # Example: review of a signup form email check that hangs ## User request "Our signup API sometimes times out and the profiler points at this line. Can you explain the regex and fix it? It runs in Python 3.12 on the email field of a public form." ```python EMAIL = re.compile(r"^([a-zA-Z0-9]+\.?)+@[A-z]+.com$") ``` ## Cases used (cases.txt) The user's 5 cases plus 3 edge cases added during the review (a dash instead of the dot, an underscore in the domain, and 30 letters with no `@`, the shape a bot sent): ``` + ann@example.com + first.last@mail.example.co - bob@@example.com - no-at-sign.example.com - trailing-dot@example. - ann@example-com - ann@exa_mple.com - aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa! ``` ## Checker run (original) ``` $ python3 scripts/regex_check.py '^([a-zA-Z0-9]+\.?)+@[A-z]+.com$' --cases cases.txt --no-explain --probe Samples (search): PASS + 'ann@example.com' -> [0:15] 'ann@example.com' 1='ann' FAIL + 'first.last@mail.example.co' -> expected a match PASS - 'bob@@example.com' PASS - 'no-at-sign.example.com' PASS - 'trailing-dot@example.' FAIL - 'ann@example-com' -> [0:15] 'ann@example-com' 1='ann' (expected no match) FAIL - 'ann@exa_mple.com' -> [0:16] 'ann@exa_mple.com' 1='ann' (expected no match) TIMEOUT - 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!' -> still running after 2 s (catastrophic backtracking) Risks and suggestions: HIGH nested-quantifier a quantifier sits inside another quantifier and nothing else in the outer repeat is required ... probe 'a' * n + '!': n=12: 0.0002 s, n=16: 0.0031 s, n=20: 0.0593 s, n=24: 0.8419 s, n=28: timeout -> TIMEOUT at length 29: exponential backtracking confirmed WARN range-A-z [A-z] also matches [ \ ] ^ _ and the backtick; use [A-Za-z] WARN unescaped-dot '.' before 'com' matches any char (so a '-' or 'x' passes there too); write \. for a literal dot INFO dollar-newline $ also matches before a trailing newline ... Summary: HIGH 1 WARN 2 INFO 1 | expectations 4/8 pass | 1 sample(s) timed out ``` ## Review (filled template, shortened) **Checker:** HIGH 1 / WARN 2 / INFO 1 before; expectations 4/8 pass (one timeout). After: HIGH 0 / WARN 0, expectations 8/8 pass. What it does: start of string; one or more runs of letters or digits, each optionally followed by a dot; an `@`; one or more chars from A to z (including `[ \ ] ^ _` and the backtick); any single char; the text `com`; end of string or before a final newline. | # | Input | Expected | Before | After | Notes | |---|---|---|---|---|---| | 1 | ann@example.com | match | match | match | | | 2 | first.last@mail.example.co | match | no match | match | only `.com` and one domain label were allowed | | 3 | bob@@example.com | no match | no match | no match | | | 4 | no-at-sign.example.com | no match | no match | no match | | | 5 | trailing-dot@example. | no match | no match | no match | | | 6 | ann@example-com | no match | match | no match | added: the unescaped dot accepted `-` | | 7 | ann@exa_mple.com | no match | match | no match | added: `[A-z]` accepted `_` | | 8 | 30 x "a" then "!" | no match, fast | timeout after 2 s | no match, instant | added: the production hang | Risks: the `([a-zA-Z0-9]+\.?)+` group lets a run of letters be split in 2^n ways when the `@` is missing, so a 29-char junk input from a bot blocked a worker for more than 2 seconds; this is a ReDoS bug on a public endpoint. Corrected pattern: ```python EMAIL = re.compile(r""" [A-Za-z0-9._%+-]+ # local part: one run, dots allowed but not repeated in a nested group @ (?:[A-Za-z0-9-]+\.)+ # one or more domain labels, each ending in a literal dot [A-Za-z]{2,} # top-level domain """, re.VERBOSE | re.ASCII) def looks_like_email(value: str) -> bool: return len(value) <= 254 and EMAIL.fullmatch(value) is not None ``` Why each change: one flat repeat for the local part removes the nested quantifier; `\.` makes the dot literal; `[A-Za-z]` replaces `[A-z]`; any top-level domain is accepted instead of only `com`; `fullmatch` replaces `^...$`; a 254-char limit is a second defense; `re.ASCII` keeps `[...]` classes predictable. Recheck: `python3 scripts/regex_check.py '[A-Za-z0-9._%+-]+@(?:[A-Za-z0-9-]+\.)+[A-Za-z]{2,}' --cases cases.txt --mode fullmatch --probe` reports no HIGH or WARN risks (the nested group is reported as INFO nested-separated, a safe shape) and 8/8 expectations pass. Keep these tests: all 8 rows above, plus a 300-char input to check the length limit. Note that a regex only checks the shape; confirm the address by sending a verification email. FILE:scripts/regex_check.py #!/usr/bin/env python3 """Explain a Python regular expression, test it on sample inputs, and warn about risky patterns. Usage: python3 regex_check.py PATTERN [options] python3 regex_check.py PATTERN -t "text one" -t "text two" python3 regex_check.py PATTERN --cases cases.txt --mode fullmatch python3 regex_check.py PATTERN --cases - < cases.txt (read cases from stdin) Options: -t, --text TEXT a sample input (repeatable) --cases FILE one sample per line; "+ text" must match, "- text" must not match, "# ..." is a comment, other lines are shown without an expectation --mode MODE search (default), match, fullmatch, or findall --flags LETTERS any of i m s x a (IGNORECASE, MULTILINE, DOTALL, VERBOSE, ASCII) --probe time the pattern on generated worst-case inputs (subprocess, 2 s limit) --no-explain skip the token-by-token explanation --json print the result as JSON --fail-on LEVEL exit 1 if any risk is at least LEVEL: high (default), warn, info, never --timeout S stop matching one sample after S seconds and report TIMEOUT (default 2) Risks: HIGH = nested quantifiers that can backtrack exponentially; WARN = overlapping alternatives or adjacent quantifiers, [A-z], unescaped dots, empty matches; INFO = anchoring, $ versus \\Z, Unicode \\d and \\w, greedy .*, flags that look missing. Exit codes: 0 ok, 1 failed expectations or a risk at or above --fail-on, 2 usage or pattern error. Python 3 standard library only (uses the re module's own parser). """ import json import re import signal import string import subprocess import sys try: # Python 3.11+ import re._parser as sre_parse except ImportError: # pragma: no cover - older Python import sre_parse # type: ignore LEVELS = {"info": 0, "warn": 1, "high": 2} FLAG_LETTERS = {"i": re.IGNORECASE, "m": re.MULTILINE, "s": re.DOTALL, "x": re.VERBOSE, "a": re.ASCII} FLAG_NAMES = [(re.IGNORECASE, "IGNORECASE"), (re.MULTILINE, "MULTILINE"), (re.DOTALL, "DOTALL"), (re.VERBOSE, "VERBOSE"), (re.ASCII, "ASCII")] ALPHABET = string.ascii_letters + string.digits + string.punctuation + " \t\n" + "\u00e9\u0663" CATEGORY_RE = {"CATEGORY_DIGIT": r"\d", "CATEGORY_NOT_DIGIT": r"\D", "CATEGORY_SPACE": r"\s", "CATEGORY_NOT_SPACE": r"\S", "CATEGORY_WORD": r"\w", "CATEGORY_NOT_WORD": r"\W"} CATEGORY_TEXT = {"CATEGORY_DIGIT": "a digit", "CATEGORY_NOT_DIGIT": "a non-digit", "CATEGORY_SPACE": "whitespace", "CATEGORY_NOT_SPACE": "a non-whitespace char", "CATEGORY_WORD": "a word char (letter, digit, _)", "CATEGORY_NOT_WORD": "a non-word char"} AT_TEXT = {"AT_BEGINNING": "^ start of string (start of each line with MULTILINE)", "AT_BEGINNING_STRING": "\\A start of string only", "AT_END": "$ end of string or before a final newline (end of each line with MULTILINE)", "AT_END_STRING": "\\Z end of string only", "AT_BOUNDARY": "\\b word boundary", "AT_NON_BOUNDARY": "\\B not a word boundary"} MAXREPEAT = sre_parse.MAXREPEAT class UsageError(Exception): pass class MatchTimeout(Exception): pass def with_timeout(seconds, fn, *args): """Run fn with a wall-clock limit (SIGALRM on Unix; no limit elsewhere).""" if not hasattr(signal, "setitimer") or seconds <= 0: return fn(*args) def handler(_sig, _frame): raise MatchTimeout() old = signal.signal(signal.SIGALRM, handler) signal.setitimer(signal.ITIMER_REAL, seconds) try: return fn(*args) finally: signal.setitimer(signal.ITIMER_REAL, 0) signal.signal(signal.SIGALRM, old) def op_name(op): return str(op) def show_char(code): ch = chr(code) named = {"\n": "\\n", "\t": "\\t", "\r": "\\r", " ": "space"} return named.get(ch, repr(ch) if ch.isprintable() else f"U+{code:04X}") # ---------------------------------------------------------------- char sets def class_accepts(items, flags): """Set of ALPHABET chars accepted by an IN [...] item list.""" negate = bool(items) and op_name(items[0][0]) == "NEGATE" acc = set() for op, av in items: name = op_name(op) if name == "LITERAL": acc.add(chr(av)) elif name == "RANGE": acc |= {c for c in ALPHABET if av[0] <= ord(c) <= av[1]} elif name == "CATEGORY": rx = re.compile(CATEGORY_RE.get(op_name(av), "."), flags & re.ASCII) acc |= {c for c in ALPHABET if rx.match(c)} if flags & re.IGNORECASE: acc |= {c.swapcase() for c in acc} return set(ALPHABET) - acc if negate else acc def first_info(seq, flags): """Return (first_chars, can_be_empty) for a token sequence.""" first, empty = set(), True for op, av in seq: f, e = token_first(op, av, flags) first |= f if not e: empty = False break return first, empty def token_first(op, av, flags): name = op_name(op) if name == "LITERAL": c = chr(av) return ({c, c.swapcase()} if flags & re.IGNORECASE else {c}), False if name == "NOT_LITERAL": return set(ALPHABET) - {chr(av)}, False if name == "IN": return class_accepts(av, flags), False if name == "ANY": return (set(ALPHABET) if flags & re.DOTALL else set(ALPHABET) - {"\n"}), False if name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): lo, _hi, sub = av f, e = first_info(sub, flags) return f, e or lo == 0 if name == "SUBPATTERN": return first_info(av[3], flags_for(av, flags)) if name == "ATOMIC_GROUP": return first_info(av, flags) if name == "BRANCH": first, empty = set(), False for b in av[1]: f, e = first_info(b, flags) first |= f empty = empty or e return first, empty if name == "GROUPREF_EXISTS": _g, yes, no = av f1, e1 = first_info(yes, flags) f2, e2 = first_info(no or [], flags) return f1 | f2, e1 or e2 if name == "GROUPREF": return set(ALPHABET), True return set(), True # AT, ASSERT, ASSERT_NOT: zero width def flags_for(av, flags): add, delete = av[1], av[2] return (flags | add) & ~delete # ---------------------------------------------------------------- explanation def quant_text(lo, hi, name): mode = {"MAX_REPEAT": "greedy", "MIN_REPEAT": "lazy", "POSSESSIVE_REPEAT": "possessive"}[name] if hi == MAXREPEAT: base = {0: "zero or more", 1: "one or more"}.get(lo, f"{lo} or more") elif lo == hi: return f"exactly {lo}" elif (lo, hi) == (0, 1): base = "optional (zero or one)" else: base = f"{lo} to {hi}" return f"{base} ({mode})" def class_text(items): negate = bool(items) and op_name(items[0][0]) == "NEGATE" parts = [] for op, av in items: name = op_name(op) if name == "LITERAL": parts.append(show_char(av)) elif name == "RANGE": parts.append(f"{show_char(av[0])}-{show_char(av[1])}") elif name == "CATEGORY": parts.append(CATEGORY_TEXT.get(op_name(av), op_name(av))) return ("any char except " if negate else "one of: ") + ", ".join(parts) def explain(seq, group_names, flags, depth=0, out=None): out = [] if out is None else out pad = " " * depth i = 0 while i < len(seq): op, av = seq[i] name = op_name(op) if name == "LITERAL": # merge runs of literals j = i while j < len(seq) and op_name(seq[j][0]) == "LITERAL": j += 1 text = "".join(chr(a) for _, a in seq[i:j]) out.append(f"{pad}the text {text!r}" if j - i > 1 else f"{pad}the char {show_char(av)}") i = j continue if name == "NOT_LITERAL": out.append(f"{pad}any char except {show_char(av)}") elif name == "IN": out.append(f"{pad}{class_text(av)}") elif name == "ANY": out.append(f"{pad}any char" + ("" if flags & re.DOTALL else " except newline")) elif name == "AT": out.append(f"{pad}{AT_TEXT.get(op_name(av), op_name(av))}") elif name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): lo, hi, sub = av out.append(f"{pad}{quant_text(lo, hi, name)} of:") explain(sub, group_names, flags, depth + 1, out) elif name == "SUBPATTERN": g = av[0] if g is None: out.append(f"{pad}non-capturing group:") else: label = f' "{group_names[g]}"' if g in group_names else "" out.append(f"{pad}capturing group {g}{label}:") explain(av[3], group_names, flags_for(av, flags), depth + 1, out) elif name == "ATOMIC_GROUP": out.append(f"{pad}atomic group (no backtracking into it):") explain(av, group_names, flags, depth + 1, out) elif name == "BRANCH": out.append(f"{pad}either:") for k, b in enumerate(av[1]): out.append(f"{pad} option {k + 1}:") explain(b, group_names, flags, depth + 2, out) elif name in ("ASSERT", "ASSERT_NOT"): direction, sub = av kind = ("lookbehind" if direction < 0 else "lookahead") neg = "negative " if name == "ASSERT_NOT" else "" out.append(f"{pad}{neg}{kind} (checks without consuming):") explain(sub, group_names, flags, depth + 1, out) elif name == "GROUPREF": out.append(f"{pad}the same text as group {av} matched") elif name == "GROUPREF_EXISTS": out.append(f"{pad}if group {av[0]} matched:") explain(av[1], group_names, flags, depth + 1, out) if av[2]: out.append(f"{pad}otherwise:") explain(av[2], group_names, flags, depth + 1, out) else: out.append(f"{pad}{name}") i += 1 return out # ---------------------------------------------------------------- risk analysis def walk_repeats(seq, flags, path=""): """Yield (op, av, flags) for every repeat node, outside lookarounds too.""" for op, av in seq: name = op_name(op) if name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): yield op, av, flags yield from walk_repeats(av[2], flags) elif name == "SUBPATTERN": yield from walk_repeats(av[3], flags_for(av, flags)) elif name == "ATOMIC_GROUP": yield from walk_repeats(av, flags) elif name == "BRANCH": for b in av[1]: yield from walk_repeats(b, flags) elif name in ("ASSERT", "ASSERT_NOT"): yield from walk_repeats(av[1], flags) def flatten(seq): """Expand groups so the body of a repeat can be read as one token list.""" out = [] for op, av in seq: name = op_name(op) if name == "SUBPATTERN" and av[1] == 0 and av[2] == 0: out.extend(flatten(av[3])) else: out.append((op, av)) return out def token_width(tok, parsed): return sre_parse.SubPattern(parsed.state, [tok]).getwidth() def all_chars(seq, flags): """Every ALPHABET char a token sequence can consume anywhere (not only first).""" acc = set() for op, av in seq: name = op_name(op) if name in ("LITERAL", "NOT_LITERAL", "IN", "ANY"): acc |= token_first(op, av, flags)[0] elif name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): acc |= all_chars(av[2], flags) elif name == "SUBPATTERN": acc |= all_chars(av[3], flags_for(av, flags)) elif name == "ATOMIC_GROUP": acc |= all_chars(av, flags) elif name == "BRANCH": for b in av[1]: acc |= all_chars(b, flags) elif name == "GROUPREF_EXISTS": acc |= all_chars(av[1], flags) | all_chars(av[2] or [], flags) elif name == "GROUPREF": acc |= set(ALPHABET) return acc def has_unbounded(tok): op, av = tok name = op_name(op) if name in ("MAX_REPEAT", "MIN_REPEAT"): return av[1] == MAXREPEAT or has_unbounded_seq(av[2]) if name == "SUBPATTERN": return has_unbounded_seq(av[3]) if name == "BRANCH": return any(has_unbounded_seq(b) for b in av[1]) return False def has_unbounded_seq(seq): return any(has_unbounded(t) for t in seq) def sample_char(chars): for pool in (string.ascii_lowercase, string.digits, string.ascii_uppercase, ALPHABET): for c in pool: if c in chars: return c return None def analyze(parsed, flags, pattern, modes_seen): risks = [] def add(level, rule, msg, probe=None): risks.append({"level": level, "rule": rule, "message": msg, **({"probe": probe} if probe else {})}) # nested quantifiers: a repeat whose body can match the same text in more than one way for op, av, fl in walk_repeats(list(parsed), flags): if op_name(op) == "POSSESSIVE_REPEAT" or av[1] <= 1: continue body = flatten(av[2]) widths = [token_width(t, parsed) for t in body] if sum(w[0] for w in widths) == sum(w[1] for w in widths): continue # fixed-width body such as (\d{4}-)+ cannot be split two ways variable = [t for t, w in zip(body, widths) if w[0] != w[1]] fixed = [t for t, w in zip(body, widths) if w[0] == w[1] and w[0] > 0] var_chars = set().union(*(all_chars([t], fl) for t in variable)) has_inner_loop = any(has_unbounded(t) for t in variable) overlap = set().union(*(all_chars([t], fl) for t in fixed)) & var_chars if fixed else set() sample = sample_char(overlap) or sample_char(var_chars) if not fixed and has_inner_loop: add("high", "nested-quantifier", "a quantifier sits inside another quantifier and nothing else in the outer repeat is " "required (the classic (a+)+ or (\\w+\\s?)+ shape), so one string can be split in " "exponentially many ways; rewrite as a single repeat, make the separator required, or use an " "atomic group or possessive quantifier (Python 3.11+)", probe=sample) elif any(not (all_chars([t], fl) & var_chars) for t in fixed): if has_inner_loop: add("info", "nested-separated", "nested quantifier with a required separator that the inner repeat cannot match " "(like ([\\w-]+\\.)+); this shape is safe") elif all(all_chars([t], fl) <= var_chars for t in fixed): add("high", "ambiguous-repeat", "the body of a repeat has variable length and every required part can also be eaten by " "the variable part (for example (x+x+)+, (aa|a)+, or (\\w+|\\d+)+), so the iterations " "can be split in exponentially many ways; merge the parts into one class or make each " "iteration start with a char the rest cannot match", probe=sample) else: add("warn", "nested-overlap", "a repeat with a variable-length body whose separator is also accepted by the inner part " "(for example ([\\w.]+\\.)+): long non-matching input can slow down sharply; make the inner " "class exclude the separator", probe=sample) # adjacent overlapping unbounded repeats in any sequence seen_pairs = set() def single_char_repeat(op, av): return (op_name(op) in ("MAX_REPEAT", "MIN_REPEAT") and av[1] == MAXREPEAT and len(av[2]) == 1 and op_name(av[2][0][0]) in ("LITERAL", "NOT_LITERAL", "IN", "ANY")) def scan_adjacent(seq, fl): toks = [t for t in flatten(seq) if op_name(t[0]) not in ("AT", "ASSERT", "ASSERT_NOT")] for (a_op, a_av), (b_op, b_av) in zip(toks, toks[1:]): if single_char_repeat(a_op, a_av) and single_char_repeat(b_op, b_av): if (id(a_av), id(b_av)) in seen_pairs: continue seen_pairs.add((id(a_av), id(b_av))) fa, _ = first_info(a_av[2], fl) fb, _ = first_info(b_av[2], fl) if fa & fb: add("warn", "adjacent-overlap", "two unbounded quantifiers in a row accept the same chars (for example \\d+\\d+ or " ".*.*), so the split between them is ambiguous and failures backtrack polynomially; " "keep one of them") for op, av in seq: name = op_name(op) if name == "SUBPATTERN": scan_adjacent(av[3], flags_for(av, fl)) elif name in ("MAX_REPEAT", "MIN_REPEAT", "POSSESSIVE_REPEAT"): scan_adjacent(av[2], fl) elif name == "BRANCH": for b in av[1]: scan_adjacent(b, fl) scan_adjacent(list(parsed), flags) # text-level checks if re.search(r"\[[^\]]*A-z", pattern): add("warn", "range-A-z", "[A-z] also matches [ \\ ] ^ _ and the backtick; use [A-Za-z]") seq = list(parsed) flat = flatten(seq) for k, (op, _av) in enumerate(flat): if op_name(op) != "ANY": continue before = flat[k - 1] if k else None after = [chr(a) for o, a in flat[k + 1:k + 4] if op_name(o) == "LITERAL"] lit_before = before is not None and op_name(before[0]) == "LITERAL" and chr(before[1]).isalnum() word_after = len(after) >= 2 and all(c.isalpha() for c in after[:2]) if (lit_before and after and after[0].isalnum()) or word_after: add("warn", "unescaped-dot", f"'.' before {''.join(after) or 'the next token'!r} matches any char (so a '-' or 'x' passes there too); " "write \\. for a literal dot") break try: if re.compile(pattern, flags).fullmatch(""): add("warn", "matches-empty", "the pattern matches the empty string, so findall/finditer return empty matches and " "validation accepts blank input; require at least one char where it matters") except re.error: pass count_dotstar = len(re.findall(r"(?<!\\)\.\*", pattern)) if count_dotstar >= 3: add("info", "many-dotstar", f"{count_dotstar} '.*' in one pattern: failures are slow on long lines; " "use negated classes like [^,]* to describe each field") elif count_dotstar and not re.search(r"\.\*\?", pattern): add("info", "greedy-dotstar", "greedy .* grabs as much as possible and then backs off; use .*? or a " "negated class such as [^\"]* when you mean 'up to the next ...'") first_ops = [op_name(op) for op, _ in seq[:1]] last = seq[-1] if seq else None anchored_start = first_ops == ["AT"] anchored_end = last is not None and op_name(last[0]) == "AT" and op_name(last[1]) in ("AT_END", "AT_END_STRING") if "fullmatch" not in modes_seen and not (anchored_start and anchored_end) and modes_seen != {"findall"}: add("info", "not-anchored", "for validation, anchor both ends (\\A ... \\Z) or use re.fullmatch; " "otherwise 'abc123xyz' passes a search for \\d+") if anchored_end and op_name(last[1]) == "AT_END" and not flags & re.MULTILINE and "fullmatch" not in modes_seen: add("info", "dollar-newline", "$ also matches before a trailing newline ('abc\\n' passes ^abc$); " "use \\Z or fullmatch for strict validation") if re.search(r"(?<!\\)\\[dw]", pattern) and not flags & re.ASCII: add("info", "unicode-classes", "\\d and \\w match Unicode digits and letters too (for example " "Arabic-Indic digits); add re.ASCII or use [0-9] / [A-Za-z0-9_] for strict input formats") return risks # ---------------------------------------------------------------- probe PROBE_CODE = r""" import re, sys, time pat, flags, s = sys.argv[1], int(sys.argv[2]), sys.argv[3] rx = re.compile(pat, flags) t = time.perf_counter(); rx.search(s); print(f"{time.perf_counter() - t:.6f}") """ def run_probe(pattern, flags, ch, compiled): tail = next((t for t in "!#\x00~ \n" if not compiled.search(ch * 3 + t)), "!") results, verdict = [], "no slowdown seen" for n in (12, 16, 20, 24, 28): s = ch * n + tail try: r = subprocess.run([sys.executable, "-c", PROBE_CODE, pattern, str(flags), s], capture_output=True, text=True, timeout=2) secs = float(r.stdout.strip() or "nan") except subprocess.TimeoutExpired: results.append({"n": n, "seconds": None}) verdict = f"TIMEOUT at length {n + 1}: exponential backtracking confirmed" break results.append({"n": n, "seconds": round(secs, 4)}) prev = results[-2]["seconds"] if len(results) >= 2 else None if prev and secs > 0.01 and secs / prev > 3: verdict = (f"time grows about {secs / prev:.0f}x per +4 chars: exponential backtracking confirmed") elif secs > 0.5: verdict = f"slow ({secs:.2f} s at length {n + 1}) but not clearly exponential" return {"input": f"{ch!r} * n + {tail!r}", "timings": results, "verdict": verdict} # ---------------------------------------------------------------- matching def read_cases(path): text = sys.stdin.read() if path == "-" else open(path, encoding="utf-8").read() cases = [] for raw in text.splitlines(): if not raw.strip() or raw.lstrip().startswith("#"): continue if raw.startswith("+ ") or raw.startswith("- "): cases.append({"text": raw[2:], "expect": raw[0] == "+"}) else: cases.append({"text": raw, "expect": None}) return cases def match_case(rx, text, mode): fn = {"search": rx.search, "match": rx.match, "fullmatch": rx.fullmatch}.get(mode) names = {v: k for k, v in rx.groupindex.items()} def groups(m): out = [] for g in range(1, rx.groups + 1): out.append({"group": g, "name": names.get(g), "value": m.group(g), "span": list(m.span(g)) if m.group(g) is not None else None}) return out if mode == "findall": ms = list(rx.finditer(text)) return {"matched": bool(ms), "matches": [{"text": m.group(0), "span": list(m.span()), "groups": groups(m)} for m in ms]} m = fn(text) if not m: return {"matched": False, "matches": []} return {"matched": True, "matches": [{"text": m.group(0), "span": list(m.span()), "groups": groups(m)}]} def describe_flags(flags): names = [n for f, n in FLAG_NAMES if flags & f] return ", ".join(names) if names else "none" # ---------------------------------------------------------------- main def parse_args(argv): opts = {"texts": [], "cases": None, "mode": "search", "flags": 0, "probe": False, "explain": True, "json": False, "fail_on": "high", "pattern": None, "timeout": 2.0} i = 0 while i < len(argv): a = argv[i] def val(): if i + 1 >= len(argv): raise UsageError(f"{a} needs a value") return argv[i + 1] if a in ("-t", "--text"): opts["texts"].append(val()); i += 2 elif a == "--cases": opts["cases"] = val(); i += 2 elif a == "--mode": opts["mode"] = val(); i += 2 if opts["mode"] not in ("search", "match", "fullmatch", "findall"): raise UsageError("--mode must be search, match, fullmatch, or findall") elif a == "--flags": letters = val(); i += 2 bad = [c for c in letters.lower() if c not in FLAG_LETTERS] if bad: raise UsageError(f"unknown flag letter(s) {''.join(bad)!r}; use any of i m s x a") for c in letters.lower(): opts["flags"] |= FLAG_LETTERS[c] elif a == "--fail-on": opts["fail_on"] = val().lower(); i += 2 if opts["fail_on"] not in ("high", "warn", "info", "never"): raise UsageError("--fail-on must be high, warn, info, or never") elif a == "--timeout": try: opts["timeout"] = float(val()) except ValueError: raise UsageError("--timeout needs a number of seconds, for example --timeout 2") i += 2 elif a == "--probe": opts["probe"] = True; i += 1 elif a == "--no-explain": opts["explain"] = False; i += 1 elif a == "--json": opts["json"] = True; i += 1 elif a in ("-h", "--help"): print(__doc__); sys.exit(0) elif a.startswith("--") and opts["pattern"] is not None: raise UsageError(f"unknown option {a}") elif opts["pattern"] is None: opts["pattern"] = a; i += 1 else: raise UsageError(f"unexpected argument {a!r} (quote the pattern; use -t for sample inputs)") if opts["pattern"] is None: raise UsageError("give a PATTERN (quote it in single quotes in the shell)") return opts def main(argv): try: opts = parse_args(argv) cases = [{"text": t, "expect": None} for t in opts["texts"]] if opts["cases"]: try: cases += read_cases(opts["cases"]) except OSError as e: raise UsageError(f"cannot read cases file: {e.strerror}: {opts['cases']}") except UsageError as e: print(f"error: {e}", file=sys.stderr) return 2 pattern, flags, mode = opts["pattern"], opts["flags"], opts["mode"] try: rx = re.compile(pattern, flags) parsed = sre_parse.parse(pattern, flags) except re.error as e: pos = f" at position {e.pos}" if e.pos is not None else "" print(f"error: pattern does not compile: {e.msg}{pos}", file=sys.stderr) if e.pos is not None: print(f" {pattern}\n {' ' * e.pos}^", file=sys.stderr) return 2 flags = int(rx.flags) & ~int(re.UNICODE) names = {v: k for k, v in rx.groupindex.items()} risks = analyze(parsed, flags, pattern, {mode}) if opts["probe"]: for r in risks: if r.get("probe"): r["probe_result"] = run_probe(pattern, flags, r["probe"], rx) results = [] for c in cases: try: res = with_timeout(opts["timeout"], match_case, rx, c["text"], mode) status = None if c["expect"] is None else ("PASS" if res["matched"] == c["expect"] else "FAIL") except MatchTimeout: res, status = {"matched": None, "matches": []}, "TIMEOUT" results.append({**c, **res, "status": status}) fails = sum(1 for r in results if r["status"] in ("FAIL", "TIMEOUT")) counts = {lvl: sum(1 for r in risks if r["level"] == lvl) for lvl in ("high", "warn", "info")} report = {"pattern": pattern, "flags": describe_flags(flags), "mode": mode, "groups": rx.groups, "group_names": {str(k): v for k, v in names.items()}, "explanation": explain(list(parsed), names, flags) if opts["explain"] else [], "cases": results, "failed_expectations": fails, "risks": risks, "risk_counts": counts} threshold = LEVELS.get(opts["fail_on"], 99) risky = any(LEVELS[r["level"]] >= threshold for r in risks) if opts["json"]: print(json.dumps(report, indent=2, ensure_ascii=False)) return 1 if fails or risky else 0 print(f"Pattern: {pattern}") named = ", ".join(f'{g} "{n}"' for g, n in sorted(names.items())) print(f"Flags: {describe_flags(flags)} | mode: {mode} | groups: {rx.groups}" + (f" ({named})" if named else "")) if opts["explain"]: print("\nExplanation:") for line in report["explanation"]: print(" " + line) if results: print(f"\nSamples ({mode}):") for r in results: mark = {True: "+", False: "-", None: " "}[r["expect"]] tag = r["status"] or ("match" if r["matched"] else "no match") line = f" {tag:<8} {mark} {r['text']!r}" if r["matched"]: parts = [] for m in r["matches"][:5]: g = " ".join(f"{x['name'] or x['group']}={x['value']!r}" for x in m["groups"]) parts.append(f"[{m['span'][0]}:{m['span'][1]}] {m['text']!r}" + (f" {g}" if g else "")) more = f" (+{len(r['matches']) - 5} more)" if len(r["matches"]) > 5 else "" line += " -> " + "; ".join(parts) + more elif r["status"] == "TIMEOUT": line += f" -> still running after {opts['timeout']:g} s (catastrophic backtracking)" elif r["status"] == "FAIL": line += " -> expected a match" if r["status"] == "FAIL" and r["matched"]: line += " (expected no match)" print(line) print("\nRisks and suggestions:") if not risks: print(" none found") for r in risks: print(f" {r['level'].upper():<4} {r['rule']:<23} {r['message']}") pr = r.get("probe_result") if pr: times = ", ".join(f"n={t['n']}: {'timeout' if t['seconds'] is None else str(t['seconds']) + ' s'}" for t in pr["timings"]) print(f" probe {pr['input']}: {times} -> {pr['verdict']}") summary = f"\nSummary: HIGH {counts['high']} WARN {counts['warn']} INFO {counts['info']}" if any(r["expect"] is not None for r in results): n = sum(1 for r in results if r["expect"] is not None) summary += f" | expectations {n - fails}/{n} pass" timeouts = sum(1 for r in results if r["status"] == "TIMEOUT") if timeouts: summary += f" | {timeouts} sample(s) timed out" print(summary) return 1 if fails or risky else 0 if __name__ == "__main__": sys.exit(main(sys.argv[1:]))