🧰 Built-ins Reference

Fimod exposes the following helpers to molds — no import needed. Deprecated helpers require explicit legacy activation as described below.

Legacy built-ins

re_* (including every _fancy variant), it_unique, it_unique_by, and it_flatten are deprecated and disabled by default. An error is raised only when a mold calls one of these helpers; the error names a replacement and the compatibility variable.

For an existing mold that requires the old behavior:

FIMOD_LEGACY_BUILTINS=1 fimod s -i data.json -m old_mold.py

Only the exact value 1 enables them. Unset, 0, or any other value leaves them disabled. Activation is silent: Fimod emits no deprecation warning. This is a process setting read by Fimod, so --env is not required and the variable is not automatically exposed in the mold's env parameter. The legacy implementations and their dependencies remain available during this transition; no removal version is scheduled yet.

Migrating regex calls

Use Monty's import re in new molds:

import re

def transform(data, **_):
    match = re.search(r"(?P<user>\w+)@(?P<domain>\w+)", data["email"])
    if match:
        return {"user": match.group("user"), "domain": match.group("domain")}
    return None

count=0 replaces all matches in both substitution APIs; a negative count replaces all in the legacy helper but nothing in native re.sub. Backslash handling also differs, so test existing replacement strings before migration. Monty 1.0.0 has limitations: callable replacements are not implemented, re.ASCII does not currently restrict \w to ASCII, and a numbered replacement followed by a suffix (r"\1_suffix") needs the explicit form r"\g<1>_suffix".

To retain captured separators, use re.finditer as in the shipped @split_tags mold:

import re

def split_with_captures(pattern, text):
    parts = []
    last_end = 0
    for match in re.finditer(pattern, text):
        parts.append(text[last_end:match.start()])
        parts.extend(match.groups())
        last_end = match.end()
    parts.append(text[last_end:])
    return parts

FIMOD_REGEX_BACKTRACK_LIMIT controls only the legacy regex helpers (default: 100,000). Native Monty regex uses its own engine limit (default: 1,000,000), unaffected by this variable.

Migrating deduplication and flattening

For string values, list(dict.fromkeys(values)) keeps the first occurrence and input order. Lists and dicts cannot be dict keys. The legacy deduplication helpers distinguish 1, True, and 1.0, and compare dicts in their key order, because their keys are JSON text. To retain that behavior for JSON-shaped data:

import json

def unique(values):
    seen = set()
    result = []
    for value in values:
        key = json.dumps(value)
        if key not in seen:
            seen.add(key)
            result.append(value)
    return result


def unique_by(rows, field):
    seen = set()
    result = []
    for row in rows:
        value = row.get(field) if isinstance(row, dict) else None
        key = json.dumps(value)
        if key not in seen:
            seen.add(key)
            result.append(row)
    return result

unique_by keeps the first record; missing fields and None share a key. The shipped @dedup_by mold implements this JSON-shaped behavior.

For recursive flattening, retain strings and dicts as whole elements:

def flatten(values):
    result = []
    for value in values:
        if isinstance(value, (list, tuple)):
            result.extend(flatten(value))
        else:
            result.append(value)
    return result

itertools.chain.from_iterable flattens only one level and iterates strings and dict keys. The legacy helpers also normalize Python objects through Fimod's JSON conversion (dates become strings, tuples become lists, and integers outside the JSON integer range become strings). The Python recipes above do not reproduce every such conversion: adapt them or enable legacy compatibility when an existing mold depends on it.

🔍 Regex functions (re_*)

Legacy reference — requires FIMOD_LEGACY_BUILTINS=1.

Powered by fancy-regex — syntax based on Rust's regex crate and Oniguruma, with lookahead, lookbehind, backreferences, and atomic groups.

Two functions for replacements — re_sub uses Python syntax (\1, \g<name>); re_sub_fancy uses fancy-regex syntax ($1, ${name}).

Match result format

re_search and re_match return a dict (or None):

{
    "match":  "full match text",
    "start":  0,     # byte offset
    "end":    5,     # byte offset
    "groups": ["group1", "group2", ...],  # numbered capture groups (1..N)
    "named":  {"name": "value", ...}      # named groups, or None if no named groups
}

groups is always present (empty list if no capture groups). named is None unless the pattern uses (?P<name>...).

Function reference

re_findall — Python-style group behaviour

re_sub / re_sub_fancy — count and syntax

Both functions replace all occurrences by default. Pass an optional count to limit substitutions.

re_sub(r"\d+", "X", "a1b2c3")       # → "aXbXcX"  (all)
re_sub(r"\d+", "X", "a1b2c3", 1)    # → "aXb2c3"  (first only)

re_sub — Python re syntax in replacements:

re_sub(r"(\w+)@(\w+)", r"\2/\1", "user@host")              # → "host/user"
re_sub(r"(?P<u>\w+)@(?P<d>\w+)", r"\g<d>/\g<u>", "a@b")   # → "b/a"

re_sub_fancy — fancy-regex syntax ($1, ${name}):

re_sub_fancy(r"(\w+)@(\w+)", "$2/$1", "user@host")         # → "host/user"
re_sub_fancy(r"(\w+)@(\w+)", "$2/$1", "user@host", 1)      # first only

re_split — captured groups included

When the pattern has capture groups, captured text is included in the result (same as Python re.split):

re_split(r"([,;])\s*", "a, b;c")    # → ["a", ",", "b", ";", "c"]
re_split(r"[,;]\s*", "a, b;c")      # → ["a", "b", "c"]  (no groups = no extras)

🗂️ Dotpath functions (dp_*)

Navigate and mutate nested structures using dot-separated paths.

Path syntax

🔁 Iteration helpers (it_*)

Convenience functions for list/dict operations. it_unique, it_unique_by, and it_flatten require FIMOD_LEGACY_BUILTINS=1; other helpers in this table remain available by default. Native Python can also handle many of these operations.

#️⃣ Hash functions (hs_*)

All functions accept a single string and return a lowercase hex string.

📝 Template functions (tpl_*)

Data→text generation using Jinja2 templates (via MiniJinja). Extends Fimod's data→data pipeline to data→text for generating configs, reports, Dockerfiles, k8s manifests, etc.

tpl_render_str(template, ctx, auto_escape=False) — Render a Jinja2 template string with a context dict. All built-in Jinja2 filters (upper, join, tojson, …), loops, conditions, and macros are available.

def transform(data, args, env, headers, **_):
    return tpl_render_str("""
FROM python:{{ python_version }}-slim
{% for pkg in packages %}
RUN uv pip install {{ pkg }}
{% endfor %}
""", data)

echo '{"python_version":"3.12","packages":["flask","requests"]}' \
  | fimod s -e 'tpl_render_str("Hello {{ name }}!", data)' --output-format txt

tpl_render_from_mold(path, ctx, auto_escape=False) — Load a .j2 file relative to the mold's directory and render it. Works with directory molds and registry molds. Enables clean separation of logic (Python) and presentation (Jinja2).

# my_mold/my_mold.py
def transform(data, args, env, headers, **_):
    tpl = args.get("template", "Dockerfile.j2")
    return tpl_render_from_mold(f"templates/{tpl}", data)

fimod s -i data.json -m ./my_mold/ --output-format txt

Set auto_escape=True when generating HTML to automatically escape <, >, &, etc.

📢 Message functions (msg_*)

Output diagnostic messages to stderr without affecting the data pipeline. All functions take a single string and return None.

Which functions produce output depends on the --quiet / --msg-level flags:

def transform(data, args, env, headers, **_):
    msg_verbose(f"Input has {len(data)} records")
    missing = [r for r in data if not r.get("email")]
    if missing:
        msg_warn(f"{len(missing)} records without email")
    msg_trace(f"First record: {data[0]}")
    return data

🛡️ Gatekeeper functions (gk_*)

Validation helpers for asserting conditions and controlling pipeline failure. Work with set_exit() — gk_fail and gk_assert set exit code to 1.

gk_assert and gk_warn use Python-style truthiness: None, False, 0, 0.0, "", [] are falsy.

def transform(data, args, env, headers, **_):
    gk_assert(data.get("version"), "missing 'version' field")
    gk_warn(len(data.get("items", [])) > 0, "items list is empty")
    if data.get("coverage", 0) < 80:
        gk_fail(f"Coverage {data['coverage']}% below 80% threshold")
    return data

🔄 Environment substitution (env_subst)

Unknown variables are left as-is (standard envsubst behavior). Only ${VAR} syntax is supported ($VAR without braces is not substituted).

def transform(data, args, env, headers, **_):
    url = env_subst("https://${HOST}:${PORT}/api", env)
    return {"url": url, "data": data}

fimod s -i data.json --env 'HOST,PORT' -m inject_url.py

🚦 Exit control

Sets the process exit code from inside a mold. code is an integer 0–255. The mold continues executing to completion after the call.

See Exit Codes for the interaction with --check.

🔀 Format control

set_input_format(name) — re-parses the output of the current step as the given format before feeding it as input to the next step. Useful with --input-format http to re-parse a string body as JSON, CSV, etc.

cast_input_format(name, value) — same as set_input_format but returns value. Useful as a single-expression one-liner when both the format hint and the return value are needed.

set_output_format(name) — overrides the final output format (like a dynamic --output-format). Also accepts "raw" for binary pass-through (HTTP downloads).

# Fetch raw HTTP response, then re-parse body as JSON
fimod s -i https://jsonplaceholder.typicode.com/todos/1 \
    --input-format http \
    -e 'cast_input_format("json", data["body"])' \
    -e 'data["title"]' --output-format txt

Supported format names for set_input_format: json, ndjson, yaml, toml, csv, txt, lines, http. set_output_format additionally accepts "raw" (binary pass-through, requires --input-format http).

🧩 Transform parameters

Reusable molds should include **_ in their transform signature:

def transform(data, **_):
    return data

Fimod passes args, env, headers, and pipeline as keyword arguments. Declare the ones you need before **_; keep **_ so unused or future keywords do not break the mold.

args

Dict of --arg name=value pairs. Empty dict {} when no --arg is passed. Untyped args arrive as strings; typed arg directives can validate and cast them before the mold runs:

# fimod: arg=threshold:int
def transform(data, args, **_):
    limit  = args["threshold"]
    prefix = args.get("prefix", "")   # with default
    return [u for u in data if u["name"].startswith(prefix) and u["age"] > limit]

env

Dict of filtered environment variables. Populated by --env PATTERN (glob patterns, comma-separated, repeatable). Empty dict {} when no --env is passed:

fimod s -i data.json --env 'HOME,USER' -e 'env["HOME"]'
fimod s -i data.json --env 'GITHUB_*' -e 'env'
fimod s -i data.json --env '*' -e 'env.get("CI", "false")'

headers

List of CSV column names when the input format is CSV with a header row. None for non-CSV input or when using --csv-no-input-header:

name,score,passed
Alice,87,true
Bob,42,false
Carol,95,true

def transform(data, args, env, headers, **_):
    # headers = ["name", "score", "passed"] for CSV, None otherwise
    if headers and "score" in headers:
        return it_sort_by(data, "score")
    return data

fimod s -i grades.csv -m sort_by_score.py