Skill Tracer
Skills › Scientific Agent Skills › markitdown › workflows.md

workflows.md

309 lines · skills/markitdown/references/workflows.md at 5f2bc03

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Practical Workflows
2
3All workflows target MarkItDown 0.1.8 and use `.markdown`, not the legacy `.text_content` alias.
4

Practical Workflows

All workflows target MarkItDown 0.1.8 and use .markdown, not the legacy .text_content alias.

Raw lines 1 to 4
5## 1. One Trusted Local Document
6
7```python
8from hashlib import sha256
9from pathlib import Path
11from markitdown import MarkItDown
13source = Path("protocol.docx").resolve(strict=True)
14output = Path("protocol.md")
16result = MarkItDown().convert_local(source)
17output.write_text(result.markdown, encoding="utf-8")
19provenance = {
20 "source": source.name,
21 "sha256": sha256(source.read_bytes()).hexdigest(),
22 "title": result.title,
23 "output": str(output),
25print(provenance)
26```
28For very large files, compute the hash incrementally rather than loading the source again.

1. One Trusted Local Document

from hashlib import sha256
from pathlib import Path

from markitdown import MarkItDown

source = Path("protocol.docx").resolve(strict=True)
output = Path("protocol.md")

result = MarkItDown().convert_local(source)
output.write_text(result.markdown, encoding="utf-8")

provenance = {
    "source": source.name,
    "sha256": sha256(source.read_bytes()).hexdigest(),
    "title": result.title,
    "output": str(output),
}
print(provenance)

For very large files, compute the hash incrementally rather than loading the source again.

Raw lines 5 to 29
30## 2. Binary Upload
32Validate size and type before conversion. Use `BytesIO` plus explicit hints:
34```python
35from io import BytesIO
37from markitdown import MarkItDown, StreamInfo
40def convert_pdf_upload(payload: bytes, filename: str) -> str:
41 max_bytes = 25 * 1024 * 1024
42 if len(payload) > max_bytes:
43 raise ValueError("PDF exceeds the 25 MiB conversion limit")
44 if not payload.startswith(b"%PDF-"):
45 raise ValueError("Payload does not have a PDF signature")
47 result = MarkItDown().convert_stream(
48 BytesIO(payload),
49 stream_info=StreamInfo(
50 extension=".pdf",
51 mimetype="application/pdf",
52 filename=filename,
53 ),
54 )
55 return result.markdown
56```
58Signature checks are only one validation layer; they do not make the parser sandboxed.

2. Binary Upload

Validate size and type before conversion. Use BytesIO plus explicit hints:

from io import BytesIO

from markitdown import MarkItDown, StreamInfo


def convert_pdf_upload(payload: bytes, filename: str) -> str:
    max_bytes = 25 * 1024 * 1024
    if len(payload) > max_bytes:
        raise ValueError("PDF exceeds the 25 MiB conversion limit")
    if not payload.startswith(b"%PDF-"):
        raise ValueError("Payload does not have a PDF signature")

    result = MarkItDown().convert_stream(
        BytesIO(payload),
        stream_info=StreamInfo(
            extension=".pdf",
            mimetype="application/pdf",
            filename=filename,
        ),
    )
    return result.markdown

Signature checks are only one validation layer; they do not make the parser sandboxed.

Raw lines 30 to 59
60## 3. Batch Directory Conversion
62```bash
63python scripts/batch_convert.py inputs/ outputs/ \
64 --recursive \
65 --extensions .pdf .docx .pptx .xlsx .csv .html \
66 --manifest outputs/manifest.json
67```
69Properties:
71- Local paths only
72- Symlinks skipped
73- Deterministic path ordering
74- Relative directories preserved
75- Output name retains the source suffix, e.g. `paper.pdf.md`
76- Existing output skipped unless `--overwrite`
77- Plugins off unless `--plugins`
78- Nonzero exit when a conversion fails
80Retry failed files after installing the missing format extra:
82```bash
83uv pip install "markitdown[pdf,docx,pptx,xlsx]==0.1.8"
84python scripts/batch_convert.py inputs/ outputs/ --recursive --overwrite
85```
87Use a fresh output directory when comparing converter versions.

3. Batch Directory Conversion

python scripts/batch_convert.py inputs/ outputs/ \
  --recursive \
  --extensions .pdf .docx .pptx .xlsx .csv .html \
  --manifest outputs/manifest.json

Properties:

  • Local paths only
  • Symlinks skipped
  • Deterministic path ordering
  • Relative directories preserved
  • Output name retains the source suffix, e.g. paper.pdf.md
  • Existing output skipped unless --overwrite
  • Plugins off unless --plugins
  • Nonzero exit when a conversion fails

Retry failed files after installing the missing format extra:

uv pip install "markitdown[pdf,docx,pptx,xlsx]==0.1.8"
python scripts/batch_convert.py inputs/ outputs/ --recursive --overwrite

Use a fresh output directory when comparing converter versions.

Raw lines 60 to 88
Runs code python scripts/batch_convert.py :63
89## 4. Literature Collection
91Input naming convention:
93```text
94Author_Year_Title.pdf
95```
97Convert and build indexes:
99```bash
100python scripts/convert_literature.py papers/ literature/ \
101 --recursive \
102 --create-index
103```
105Organize by inferred year:
107```bash
108python scripts/convert_literature.py papers/ literature/ \
109 --recursive \
110 --organize-by-year \
111 --create-index
112```
114Each output contains provenance front matter:
116```yaml
117---
118title: "Example title"
119author: "Smith"
120year: "2025"
121source: "incoming/Smith_2025_Example_Title.pdf"
122converted_at: "2026-10-01T17:00:00+00:00"
123markitdown_version: "0.1.8"
124---
125```
127The helper also writes `catalog.json` and `INDEX.md` when `--create-index` is requested. Metadata inferred from a filename is a convenience, not authoritative bibliographic metadata.

4. Literature Collection

Input naming convention:

Author_Year_Title.pdf

Convert and build indexes:

python scripts/convert_literature.py papers/ literature/ \
  --recursive \
  --create-index

Organize by inferred year:

python scripts/convert_literature.py papers/ literature/ \
  --recursive \
  --organize-by-year \
  --create-index

Each output contains provenance front matter:

---
title: "Example title"
author: "Smith"
year: "2025"
source: "incoming/Smith_2025_Example_Title.pdf"
converted_at: "2026-10-01T17:00:00+00:00"
markitdown_version: "0.1.8"
---

The helper also writes catalog.json and INDEX.md when --create-index is requested. Metadata inferred from a filename is a convenience, not authoritative bibliographic metadata.

Raw lines 89 to 128
Runs code python scripts/convert_literature.py :100
129## 5. RAG Ingestion
131Recommended sequence:
1331. Validate and hash the source.
1342. Convert with a pinned package/plugin version.
1353. Keep source metadata separate from converted text.
1364. Check non-empty output and expected section markers.
1375. Sanitize active HTML and unsafe links if Markdown will be rendered.
1386. Chunk by semantic headings with overlap.
1397. Store source/hash/converter/page-or-section provenance with every chunk.
1408. Mark chunks as untrusted source content in the retrieval prompt.
1419. Retain the original source for verification.
143Example conversion envelope:
145```python
146from dataclasses import asdict, dataclass
147from hashlib import sha256
148from importlib.metadata import version
149from pathlib import Path
151from markitdown import MarkItDown
154@dataclass(frozen=True)
155class ConvertedDocument:
156 source_name: str
157 source_sha256: str
158 converter_version: str
159 title: str | None
160 markdown: str
163def convert_for_ingestion(path: Path) -> ConvertedDocument:
164 path = path.resolve(strict=True)
165 digest = sha256(path.read_bytes()).hexdigest()
166 result = MarkItDown().convert_local(path)
167 if not result.markdown.strip():
168 raise ValueError(f"Conversion produced no text: {path.name}")
170 return ConvertedDocument(
171 source_name=path.name,
172 source_sha256=digest,
173 converter_version=version("markitdown"),
174 title=result.title,
175 markdown=result.markdown,
176 )
177```
179Do not treat Markdown line numbers as stable page citations. MarkItDown does not expose PDF page coordinates.

5. RAG Ingestion

Recommended sequence:

  1. Validate and hash the source.
  2. Convert with a pinned package/plugin version.
  3. Keep source metadata separate from converted text.
  4. Check non-empty output and expected section markers.
  5. Sanitize active HTML and unsafe links if Markdown will be rendered.
  6. Chunk by semantic headings with overlap.
  7. Store source/hash/converter/page-or-section provenance with every chunk.
  8. Mark chunks as untrusted source content in the retrieval prompt.
  9. Retain the original source for verification.

Example conversion envelope:

from dataclasses import asdict, dataclass
from hashlib import sha256
from importlib.metadata import version
from pathlib import Path

from markitdown import MarkItDown


@dataclass(frozen=True)
class ConvertedDocument:
    source_name: str
    source_sha256: str
    converter_version: str
    title: str | None
    markdown: str


def convert_for_ingestion(path: Path) -> ConvertedDocument:
    path = path.resolve(strict=True)
    digest = sha256(path.read_bytes()).hexdigest()
    result = MarkItDown().convert_local(path)
    if not result.markdown.strip():
        raise ValueError(f"Conversion produced no text: {path.name}")

    return ConvertedDocument(
        source_name=path.name,
        source_sha256=digest,
        converter_version=version("markitdown"),
        title=result.title,
        markdown=result.markdown,
    )

Do not treat Markdown line numbers as stable page citations. MarkItDown does not expose PDF page coordinates.

Raw lines 129 to 180
181## 6. Mixed-Format Study Folder
183Use a two-pass workflow:

6. Mixed-Format Study Folder

Use a two-pass workflow:

Raw lines 181 to 184
185### Pass A: inventory
187- Count files by extension and byte size.
188- Detect duplicates by content hash.
189- Exclude temporary, lock, and hidden files.
190- Identify encrypted/unsupported inputs.
191- Classify which files require external OCR/cloud processing.

Pass A: inventory

  • Count files by extension and byte size.
  • Detect duplicates by content hash.
  • Exclude temporary, lock, and hidden files.
  • Identify encrypted/unsupported inputs.
  • Classify which files require external OCR/cloud processing.
Raw lines 185 to 192
193### Pass B: convert
195- Use local built-ins first.
196- Route only failed scanned/complex files to an approved OCR/cloud path.
197- Record the selected mode per file.
198- Compare file count, failure count, and output count.
200This minimizes cost and external data transfer.

Pass B: convert

  • Use local built-ins first.
  • Route only failed scanned/complex files to an approved OCR/cloud path.
  • Record the selected mode per file.
  • Compare file count, failure count, and output count.

This minimizes cost and external data transfer.

Raw lines 193 to 201
202## 7. Spreadsheet Orientation, Then Structured Analysis
204Use MarkItDown to understand workbook organization:
206```python
207from markitdown import MarkItDown
209preview = MarkItDown().convert_local("assay-results.xlsx").markdown
210print(preview)
211```
213Then use a spreadsheet/dataframe library for calculations:
215- Inspect formulas and cached values explicitly.
216- Select the authoritative sheet/range.
217- Preserve data types.
218- Validate merged cells and hidden rows.
220Do not parse a scientific numeric dataset back out of Markdown when the original workbook is available.

7. Spreadsheet Orientation, Then Structured Analysis

Use MarkItDown to understand workbook organization:

from markitdown import MarkItDown

preview = MarkItDown().convert_local("assay-results.xlsx").markdown
print(preview)

Then use a spreadsheet/dataframe library for calculations:

  • Inspect formulas and cached values explicitly.
  • Select the authoritative sheet/range.
  • Preserve data types.
  • Validate merged cells and hidden rows.

Do not parse a scientific numeric dataset back out of Markdown when the original workbook is available.

Raw lines 202 to 221
222## 8. Presentation Review
224```python
225from markitdown import MarkItDown
227result = MarkItDown().convert_local("lab-meeting.pptx")
228```
230Review:
232- Slide boundaries
233- Title hierarchy
234- Speaker notes
235- Text in charts/images
236- Claims encoded only by color/position
238If images carry essential meaning, add an approved vision description or OCR pass and label generated text as model-derived.

8. Presentation Review

from markitdown import MarkItDown

result = MarkItDown().convert_local("lab-meeting.pptx")

Review:

  • Slide boundaries
  • Title hierarchy
  • Speaker notes
  • Text in charts/images
  • Claims encoded only by color/position

If images carry essential meaning, add an approved vision description or OCR pass and label generated text as model-derived.

Raw lines 222 to 239
240## 9. Scanned-PDF Escalation
2421. Try built-in PDF extraction.
2432. Detect unexpectedly short/empty output relative to page/file size.
2443. Confirm the source is scanned by visual inspection.
2454. Choose:
246 - Local OCR/layout parser for restricted data
247 - `markitdown-ocr` for approved vision-provider processing
248 - Document Intelligence for Azure layout/OCR
249 - Content Understanding for structured fields/multimodal processing
2505. Validate a page sample and record the selected service/model/analyzer.
252Do not automatically send failed local conversions to a cloud service.

9. Scanned-PDF Escalation

  1. Try built-in PDF extraction.
  2. Detect unexpectedly short/empty output relative to page/file size.
  3. Confirm the source is scanned by visual inspection.
  4. Choose:
  5. Local OCR/layout parser for restricted data
  6. markitdown-ocr for approved vision-provider processing
  7. Document Intelligence for Azure layout/OCR
  8. Content Understanding for structured fields/multimodal processing
  9. Validate a page sample and record the selected service/model/analyzer.

Do not automatically send failed local conversions to a cloud service.

Raw lines 240 to 253
254## 10. Application-Controlled Remote Resource
256The application—not MarkItDown—should:
258- Validate destination and every redirect
259- Enforce time and byte limits
260- Restrict MIME types
261- Apply authentication safely
262- Produce a bounded `requests.Response`
264Then:
266```python
267result = MarkItDown().convert_response(validated_response)
268```
270See `security.md` for the complete SSRF policy. A simple scheme check is not sufficient.

10. Application-Controlled Remote Resource

The application—not MarkItDown—should:

  • Validate destination and every redirect
  • Enforce time and byte limits
  • Restrict MIME types
  • Apply authentication safely
  • Produce a bounded requests.Response

Then:

result = MarkItDown().convert_response(validated_response)

See security.md for the complete SSRF policy. A simple scheme check is not sufficient.

Raw lines 254 to 271
272## 11. Deterministic Regression Corpus
274Keep a small, redistributable corpus covering:
276- Text-native and scanned PDFs
277- DOCX headings/tables/equations
278- PPTX notes/grouped shapes
279- XLSX multiple sheets/merged cells
280- CSV quoted fields and Unicode
281- HTML links/lists/tables
282- EPUB chapters
283- JPEG/PNG metadata
285For each fixture, assert:
287- Conversion succeeds or fails with the expected category.
288- Required headings/sentinel text are present.
289- No source is unexpectedly routed to network/cloud.
290- Output is UTF-8 and below a reasonable size.
291- Package/plugin versions are recorded.
293Avoid asserting a full byte-for-byte Markdown snapshot unless exact formatting stability is required; semantic assertions are less brittle.

11. Deterministic Regression Corpus

Keep a small, redistributable corpus covering:

  • Text-native and scanned PDFs
  • DOCX headings/tables/equations
  • PPTX notes/grouped shapes
  • XLSX multiple sheets/merged cells
  • CSV quoted fields and Unicode
  • HTML links/lists/tables
  • EPUB chapters
  • JPEG/PNG metadata

For each fixture, assert:

  • Conversion succeeds or fails with the expected category.
  • Required headings/sentinel text are present.
  • No source is unexpectedly routed to network/cloud.
  • Output is UTF-8 and below a reasonable size.
  • Package/plugin versions are recorded.

Avoid asserting a full byte-for-byte Markdown snapshot unless exact formatting stability is required; semantic assertions are less brittle.

Raw lines 272 to 294
295## 12. Conversion Quality Report
297Track per file:
299- Source identifier and hash
300- Bytes and extension
301- Converter mode
302- MarkItDown/plugin version
303- Output characters/lines
304- Optional title
305- Warning/failure category
306- Manual validation status
307- External provider/analyzer, if any
309The bundled batch manifest provides a baseline for these records.

12. Conversion Quality Report

Track per file:

  • Source identifier and hash
  • Bytes and extension
  • Converter mode
  • MarkItDown/plugin version
  • Output characters/lines
  • Optional title
  • Warning/failure category
  • Manual validation status
  • External provider/analyzer, if any

The bundled batch manifest provides a baseline for these records.

Raw lines 295 to 309