Coverage for src/local_deep_research/document_loaders/zip_container_guard.py: 97%
1856 statements
« prev ^ index » next coverage.py v7.16.2, created at 2026-10-22 21:20 +0000
1"""Decompression-bomb guard for zip-container document uploads.
3``.docx``/``.xlsx``/``.pptx``/``.odt``/``.epub`` files are zip
4containers of XML parts, while the upload route's caps bound only the
5*compressed* request body. A modest container can therefore declare —
6or, with lying headers, secretly contain — gigabytes of XML and OOM
7the indexing worker: the same class as the OpenAlex partition caps and
8the PDF extraction ceilings.
10What the guard checks, before any parser sees the bytes:
120. **Anchoring**: the upload starts with a zip local file header (or,
13 for an empty archive, the end-of-central-directory record) and the
14 first entry's local header sits at offset 0. zipfile locates the
15 central directory from the *end* of the file and tolerates data in
16 front of the archive, so without this a legacy OLE ``.doc``/``.ppt``
17 with a small zip appended would pass as ``.docx``/``.pptx`` — and
18 unstructured, which sniffs the leading bytes, would then route it to
19 its own unbounded soffice conversion.
201. **Declarations** (central directory only, no decompression): a
21 declared central-directory byte span of at most ``max_entries``
22 kibibytes, read from the end-of-central-directory (and ZIP64)
23 records before ``zipfile`` walks them — zipfile materialises every
24 record before the entry ceiling can count them, and the EOCD's
25 record count wraps at 65,535 and is ignored by zipfile, so only
26 the declared span bounds the walk up front; at
27 most ``MAX_ENTRIES`` entries; only STORED or DEFLATE compression
28 (CPython's zipfile decompresses BZIP2/LZMA/ZSTD chunks with no
29 output limit); no encrypted entries; a STORED entry's compressed
30 size equals its declared size and a DEFLATE entry's is within a
31 small slack of it (a full read copies the whole compressed range);
32 no two entries share a local header; every member name is
33 canonical (no empty, ``.`` or ``..`` segments, so no ``//``, ``/./``
34 or leading ``/``; no backslash) and unique, because readers
35 normalise such names onto one path (zipfile itself truncates a name
36 at NUL, so NUL-truncated names that collide are caught as
37 duplicates); no entry declares more than
38 the largest per-entry ceiling; and at most ``MAX_CONTAINER_BYTES``
39 declared across all entries.
402. **Honesty of the declarations**: every entry's local header must
41 agree with its central-directory record (method, flags, CRC, both
42 sizes, name; with a data descriptor, the descriptor must repeat the
43 central CRC and sizes), because pandoc's zip reader follows the
44 local header where zipfile follows the central directory. Every
45 DEFLATE entry is then inflated once, in bounded steps (a few MiB of
46 memory, at most the declared total of work), and refused unless the
47 stream ends exactly at the end of its compressed range having
48 produced exactly the declared size with the declared CRC; a STORED
49 entry's size is fixed by its equality check in step 1 and its CRC is
50 checked too. zipfile only
51 truncates to the declared size *after* inflating a chunk, and a
52 full ``read()`` inflates an entry's whole compressed range in one
53 step, so without this a DEFLATE entry declaring 100 bytes could
54 still materialize gigabytes inside a parser.
553. **Per-kind ceilings** on the verified sizes: a parsed part (XML or
56 any non-media entry) at most ``MAX_ENTRY_BYTES`` and parsed parts at
57 most ``MAX_TOTAL_BYTES`` in total; a binary media entry at most
58 ``MAX_MEDIA_ENTRY_BYTES`` — media ceilings apply to
59 ``.docx``/``.pptx``/``.xlsx`` only (``MEDIA_CEILING_EXTENSIONS``).
60 Media classification is described below.
614. **Sheet fan-out** (``.xlsx`` only): openpyxl converts a sheet part
62 once per ``<sheet>`` that resolves to it, so a workbook whose sheets
63 resolve to the same part more than once is refused. The sheet list
64 comes from openpyxl's own ``ExcelReader`` / ``WorkbookParser`` (the
65 steps and options ``load_workbook`` uses under pandas), not from a
66 re-implementation, so the guard cannot disagree with the loader
67 about tags, attributes, the workbook's location, relationship
68 resolution (including ``TargetMode="External"`` targets, which
69 openpyxl keeps unresolved) or XML decoding. Sheets must also have
70 distinct relationships parts as computed by openpyxl's
71 ``get_rels_path``, which openpyxl parses once per sheet. Only
72 ``[Content_Types].xml``, the workbook part and its relationships are
73 parsed, each first held to ``MAX_XLSX_SKELETON_PART_BYTES`` so this
74 extra full-tree parse stays cheap; any openpyxl failure there is
75 refused as malformed. A comment or processing instruction below the
76 root's children of one of these parts or of ``xl/styles.xml`` is
77 refused (``_refuse_xlsx_nested_node``): openpyxl builds every child
78 node of a list such as ``fonts`` or ``bookViews`` into an entry, while
79 the guard counts elements. openpyxl also builds an object from every
80 entry of every list of the styles and workbook parts and from every
81 child it knows of every other element (keeping the last of a
82 repeated one), so each list of the two parts is capped
83 (``_xlsx_list_cap``; the workbook's defined names as
84 ``_check_xlsx_defined_names`` holds them), as are the elements under
85 each root (``MAX_XLSX_TOP_LEVEL_ELEMENTS``), the elements under
86 ``dxfs``, the gradient stops in all and the elements of one font,
87 fill, border or cell-format entry (``MAX_XLSX_STYLE_ENTRY_ELEMENTS``),
88 and an element that is not a list may not repeat a child's local
89 name (``_refuse_xlsx_repeated_child``). openpyxl parses the print
90 titles and print areas among the defined names with regular
91 expressions in time quadratic in their length, on every load, so
92 each defined name is held to ``MAX_XLSX_DEFINED_NAME_CHARS`` and
93 the print names together to ``MAX_XLSX_PRINT_NAME_WORK``. A workbook
94 listing more than ``MAX_XLSX_SHEETS`` sheets is refused: openpyxl
95 and pandas do work per sheet that grows with the members and with
96 the sheets; so is one with a sheet name over
97 ``MAX_XLSX_SHEET_NAME_CHARS`` characters or two sheets of one name
98 (lowercased). Before openpyxl parses any part, and before any part
99 the guard parses, each part openpyxl's read-only load parses (found
100 as it finds them: ``[Content_Types].xml``, the workbook part and its
101 relationships, then the shared strings the manifest names, the core
102 and custom document properties, the styles part, and every sheet's
103 resolved part and relationships part; ``_xlsx_loaded_parts``) is
104 streamed once with the parser that load uses for it, lxml or expat
105 (``[Content_Types].xml`` with both, for msoffcrypto's minidom), and
106 no other member is (``_check_xlsx_namespaces``): an encoding other
107 than UTF-8 (``_require_utf8_xml``; the parsers detect and decode
108 others differently, and the guard counts bytes), a namespace URI over ``MAX_XLSX_NAMESPACE_URI_CHARS`` characters,
109 which lxml and expat copy into the name of every element under it,
110 a DTD, nesting deeper than ``MAX_XLSX_XML_DEPTH``, a start tag of
111 more than ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` attributes or more than
112 ``MAX_XLSX_PART_NAMES`` distinct names (state both parsers keep per
113 open element, attribute and name) is refused, as is an lxml part
114 that lxml fails to parse (openpyxl's load fails on it too); an expat
115 part is read up to its first syntax error, where the loader's parse
116 ends. The
117 styles part ``xl/styles.xml``, which openpyxl loads on every read,
118 is streamed once (``_check_xlsx_styles``): a number format code over
119 ``MAX_XLSX_NUMBER_FORMAT_CHARS`` characters, a cell-format list of
120 more than ``MAX_XLSX_CELL_FORMATS`` entries, a gradient fill of more
121 than ``MAX_XLSX_GRADIENT_STOPS`` stops, more than
122 ``MAX_XLSX_STYLE_LIST_ENTRIES`` fonts, fills or borders, a numeric
123 field outside 32 bits, more than ``MAX_XLSX_STYLE_HASH_CHAIN``
124 distinct style objects of one hash in an indexed list openpyxl
125 builds (each entry is rebuilt with openpyxl's classes to compute
126 it), or a format code, cell-format
127 id or named-style field given as a child element (which openpyxl
128 reads in place of the attribute) is refused, as is a stylesheet
129 whose date tests over the cell formats' codes exceed
130 ``MAX_XLSX_NUMBER_FORMAT_WORK`` or whose named styles, each priced
131 at a fixed cost plus the custom formats and the largest font, fill
132 and border it can bind, exceed ``MAX_XLSX_NAMED_STYLE_WORK``
133 (openpyxl repeats both per cell format or named style).
1345. **Slide and section fan-out** (``.pptx``/``.docx``): a presentation
135 listing one slide part under several ``<p:sldId>`` entries is
136 refused, as is one whose slide placeholders would cost python-pptx's
137 uncached layout and master lookups more than
138 ``MAX_PPTX_PLACEHOLDER_WORK`` (each slide's placeholders times the
139 shapes, nodes and relationships one inherited position read walks;
140 every slide is streamed once for this, as is every slide layout and
141 slide master a slide with placeholders inherits from). Every
142 relationships part of a presentation or document is streamed and,
143 like every other part the guard streams or parses from a
144 presentation or document, first held to UTF-8 and to start tags of
145 at most ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` attributes, counted in its
146 bytes before lxml reads them (``_CountedXmlStream``; lxml's
147 ``iterparse`` builds a start tag's element in full, with no limit,
148 before any per-element ceiling sees it), then held to ``MAX_PART_RELATIONSHIPS`` entries (comments and processing
149 instructions included; an entry with child elements, or any element
150 with more than ``MAX_RELS_ELEMENT_ATTRIBUTES`` attributes, is
151 refused) and all of them together to
152 ``MAX_PACKAGE_RELATIONSHIPS``, and a document's internal
153 relationships times its members to ``MAX_DOCX_RELATIONSHIP_WALK``
154 (python-docx scans a list of the parts it has visited once per
155 relationship as it opens the package). The parts the guard reaches through
156 relationships (the main part; a presentation's slides and the
157 layouts and masters it prices; a document's styles, settings,
158 header and footer parts) must carry the content type, resolved from
159 ``[Content_Types].xml`` as the libraries resolve it, that the
160 library builds such a part from: python-pptx and python-docx pick a
161 part's class by content type, and a slide or document part standing
162 in for a layout, styles or settings part makes the library follow
163 that part's own relationship to one the guard never priced (see
164 ``_require_content_type``). A document is refused when it has more than
165 ``MAX_DOCX_SECTIONS`` ``w:sectPr`` elements or more than
166 ``MAX_DOCX_PAGE_BREAKS`` ``w:lastRenderedPageBreak`` elements, when its
167 settings or styles part is over
168 ``MAX_OPC_SKELETON_PART_BYTES`` or has more than
169 ``MAX_DOCX_SETTINGS_CHILDREN`` or ``MAX_DOCX_STYLES`` top-level
170 children, when a table cell spans, or a row skips, more than
171 ``MAX_DOCX_GRID_SPAN`` grid columns, when a table row has more than
172 ``MAX_DOCX_ROW_CELLS`` cells or spans more than
173 ``MAX_DOCX_ROW_GRID_COLUMNS`` columns, when a table row has more
174 than ``MAX_DOCX_ROW_OTHER_CHILDREN`` other children or a table more
175 than ``MAX_DOCX_TABLE_ROW_GAP`` children between (or before) two of
176 its rows, when an element of
177 ``document.xml`` carries more than ``MAX_DOCX_ELEMENT_ATTRIBUTES``
178 attributes or sees more than ``MAX_DOCX_NAMESPACES_IN_SCOPE``
179 namespace declarations in scope, when a body-level paragraph holds
180 more than ``MAX_DOCX_PARAGRAPH_NAMESPACES`` declarations, or when a
181 declaration's URI or prefix is longer than
182 ``MAX_DOCX_NAMESPACE_URI_CHARS`` or ``MAX_DOCX_NAMESPACE_PREFIX_CHARS``
183 (abnormal input no writer produces, refused outright rather than
184 priced), or when python-docx's and
185 unstructured's repeated work, estimated from one streaming pass over
186 ``document.xml``, exceeds a ceiling: the section lookup over the
187 body (``MAX_DOCX_SECTION_BLOCK_WORK``, ``MAX_DOCX_SECTION_WORK``),
188 the header/footer parts its sections reference, counted per
189 reference with the node-set unions each read evaluates
190 (``MAX_DOCX_HEADER_FOOTER_BYTES``), the style lookups per
191 paragraph (``MAX_DOCX_STYLE_WORK``), the splitting of paragraphs at
192 rendered page breaks (``MAX_DOCX_PAGE_BREAK_WORK``), the text
193 extraction from runs and table cells, once per spanned column, with
194 each cell's paragraphs and tables, the node-set unions python-docx
195 evaluates over a cell's, a paragraph's and a run's children (and,
196 in a document with no sections, the body's; and unstructured's
197 page-break union over the whole body and its paragraph-text union,
198 whose third branch reaches runs in inline drawings) and
199 python-docx's steps up vertical merges (``MAX_DOCX_RUN_WORK``), and
200 the bytes copied per page break,
201 spanned column and continued merged cell
202 (``MAX_DOCX_COPIED_BYTES``). These are the fan-outs that turn one
203 bounded part or element into many times its work or memory; each
204 ceiling holds its fan-out to a measured order of magnitude (about
205 45 s, or a few hundred MB, at the slowest layout found), and the
206 seven share one budget: the work's shares of their ceilings may sum
207 to at most 1 (see ``_check_pptx_slide_fan_out`` and
208 ``_check_docx_section_fan_out``). They do not bound a document's
209 total parse time or memory (see "Known gaps").
2106. **Package type** (``.docx``/``.pptx``/``.xlsx``, and the
211 ``mimetype`` of ``.odt``/``.epub``): langchain's Word and PowerPoint
212 loaders ask unstructured's ``detect_filetype`` for the type (when
213 python-magic imports) and send a file it calls DOC or PPT to soffice
214 with no timeout, whatever the extension. An OOXML package is
215 therefore refused when it has a root ``mimetype`` member, when its
216 member names would not make the detector name its extension, when
217 an ``.xlsx`` has no ``xl/workbook.xml`` (pandas would pick another
218 engine), or when a ``.docx``/``.pptx`` has no main document part as
219 python-docx/python-pptx resolve it; an ``.odt``/``.epub`` whose
220 ``mimetype`` member names another type is refused too (see
221 ``_check_package_type``).
223Containers the stdlib cannot list or read, and spreadsheet packages
224openpyxl cannot resolve, are refused as ``DecompressionBombError``
225(fail closed) for the failure types caught in
226``validate_zip_container`` and ``_check_xlsx_sheet_fan_out``.
228What this bounds, and what it does not:
230- **In-process Python parsers** (python-docx, python-pptx, openpyxl,
231 unstructured for ``.docx``/``.xlsx``/``.pptx``): each *read* of an
232 entry now yields at most its (verified) declared size. It does not
233 bound parser memory or parser time. For an XML parse tree alone the
234 cost is a multiple of the XML size (around 10x measured for
235 text-heavy ``.docx``, more for many small elements), which is why the
236 parsed-part ceilings sit well under a worker's memory; but what the
237 loaders build *from* that tree can be far larger than the XML and is
238 not bounded here (see "Known gaps" below). python-docx/-pptx parse
239 with lxml ``resolve_entities=False``, openpyxl uses its safe parser,
240 and libxml2/expat enforce their own amplification limits, so entities
241 are not expanded; but an unexpanded entity reference stays in
242 python-docx's tree as a node its per-section XPath walks, and the
243 streaming count of that work (step 5) sees no event for it. Every XML
244 part the guard parses or streams (OPC relationships,
245 ``presentation.xml``, the slides and the slide layouts and masters
246 their placeholders inherit from, ``document.xml``, the ``.docx`` settings part
247 and the header/footer parts its sections reference, and the ``.xlsx`` parts
248 openpyxl's reader parses for the sheet check and ``xl/styles.xml``) is
249 therefore refused if it carries a
250 document type declaration, which is the only way to declare an
251 entity; OPC (ECMA-376 Part 2) forbids DTDs in the parts it defines
252 (content types, relationships, core properties), and Office writes
253 none in the XML parts it generates (an SVG image it stores may carry
254 one, but no loader parses images). The declaration is refused in the
255 part's bytes, before any parser the guard drives receives it
256 (``_XmlProlog``): libxml2 reads a whole internal subset before the
257 root element's first event, and the attribute defaults it declares
258 add attributes to start tags whose bytes show none. Parts the guard
259 does not read are not checked. How *often* a parser reads an entry is bounded only
260 where a fan-out is known and closed: shared sheet parts and
261 per-format stylesheet work in ``.xlsx`` (step 4), repeated slides and per-section, per-paragraph, per-page-
262 break and per-spanned-column work and header/footer repeats in
263 ``.pptx``/``.docx`` (step 5), and openpyxl's per-reference drawing
264 reads (below).
265- **Binary media** (images, audio/video, fonts, embedded packages):
266 python-pptx and python-docx load each package part once, however
267 many times it is referenced, and keep it as an opaque blob — about
268 1x its size (measured for one and for eight references to the same
269 image). openpyxl would instead re-read a chartsheet drawing's image
270 once per reference and keep every copy; the ``.xlsx`` path disables
271 openpyxl's drawing reader (see ``openpyxl_hardening``; in the
272 read-only mode pandas uses, chartsheet drawings are the only
273 drawings openpyxl reads), so ``.xlsx`` media is not read at all.
274 Only an entry of an OOXML container whose first bytes carry a known
275 binary signature earns the media ceiling — never a name: part names
276 are attacker-chosen and packages dispatch parts by content type.
277 Each signature is anchored so that no XML parser the in-process
278 loaders use (lxml/expat) accepts it as a document start. EPUB and
279 ODT entries never earn it: pandoc reads an EPUB chapter by its
280 declared media type, not its bytes. All but two
281 (ISO BMFF and EMF) are fixed prefixes that start with neither
282 ``<``, whitespace nor a complete BOM (JPEG's ``FF D8`` shares only
283 its first byte with the ``FF FE`` UTF-16/32 BOM). ISO BMFF requires
284 two leading zero bytes, a big-endian box size of at least 8 and
285 ``ftyp`` (unparseable as XML in any encoding: ``ftyp`` read as UCS-4
286 is above U+10FFFF); EMF requires record type 1 (``01 00 00 00``) and
287 `` EMF`` at offset 40. As a second line, any other entry whose head
288 decodes to markup (``<`` after an optional BOM and whitespace, in
289 UTF-8/16/32 with or without a BOM) stays a parsed part whatever else
290 it matches. A
291 markup-looking name also always keeps an entry under the parsed-part
292 ceilings.
293- **pandoc** (``.odt``/``.epub``, see ``utilities/pandoc_conversion``)
294 has its own zip reader, which locates entries through the central
295 directory but takes each entry's method and compressed size from its
296 local header; step 2's header agreement makes it inflate the streams
297 verified here. Reference fan-out inside pandoc's readers is not
298 checked; the pandoc child is bounded by its own heap cap and timeout.
300Known gaps: pre-existing parser behaviour this guard does **not**
301bound. It bounds container and entry sizes, compression, the honesty
302of the declarations, and only the specific fan-outs listed above.
304- **.xlsx grid padding.** Memory follows the *cell references*, not the
305 XML size. pandas' openpyxl reader (``_OpenpyxlReader.get_sheet_data``,
306 which unstructured's ``partition_xlsx`` reaches through
307 ``pd.read_excel``) pads every row to the widest row
308 (``data_row + (max_width - len(data_row)) * empty_cell``), so a
309 few-KB sheet with cells ``XFD1`` and ``A300`` materialises
310 300 x 16384 cells (about +121 MiB RSS measured), and ``XFD1`` with
311 ``A1048576`` would need hundreds of GiB. The per-part ceilings do
312 not limit this. Likewise, a shared string is stored once in the
313 package but referenced by index from any number of cells, and
314 unstructured renders each sheet with ``DataFrame.to_html``, which
315 writes the string out once per referencing cell, so a long shared
316 string referenced by many cells multiplies into the HTML.
317- **Repeated references inside pandoc's readers** (for example an EPUB
318 spine listing one chapter many times) are not counted; pandoc is
319 bounded only by its heap cap and timeout.
320- **.docx total parse time and memory.** The ceilings in step 5
321 bound the known fan-outs, not the document: python-docx's and
322 unstructured's cost stays linear in the element count and in the
323 text, with a large constant. A trivial paragraph costs about
324 0.6-2 ms (5,000 one-letter paragraphs under python-docx's template
325 took 11 s); python-docx-built paragraphs of ordinary text 3-9 ms
326 (11,300 one-run paragraphs in one section took 31 s; 5,600
327 paragraphs of seven formatted runs beside 300 styles, 49 s); text
328 that triggers unstructured's per-element classification 35-65 ms
329 (a US city/state/ZIP regex backtracks for ~40 ms on text starting
330 with 80 or more letters without a space or comma, and langdetect
331 runs once per element at ~25 ms when the whole text is not one
332 language), so an accepted document of a few thousand such
333 paragraphs runs for minutes; and one 3 MB paragraph took 43 s and
334 ~300 MB. Nothing here bounds that; only a wall-clock and memory
335 limit on the in-process parse would.
336- **.docx length refusals.** The ceilings share one budget and price
337 each step at the slowest layout measured, so they also refuse long
338 ordinary documents that would partition in under a minute. Measured
339 on documents python-docx writes: a single section of more than
340 about 11,300 one-run paragraphs; a Word-like document (300 styles,
341 five sections, a table) of more than about 5,250-5,850 paragraphs of
342 5-10 formatted runs; more than about 580 one-paragraph sections, or
343 180 ten-paragraph ones (a mail merge to "individual documents").
344 Long theses and books can exceed these. Vertical merges are priced
345 the same way: one column merged down more than about 545 rows is
346 refused (300 such rows took 8 s to partition), as is every cell of
347 a 64-column table continuing down 27 rows (26 such rows took 22 s),
348 or down 26 with cell widths set.
349- **.pptx length refusals.** ``MAX_PPTX_PLACEHOLDER_WORK`` counts every
350 slide placeholder as inheriting its position and prices each read at
351 the slowest layout measured, so it also refuses long ordinary decks:
352 about 2,600 title-and-content slides under Office's default theme,
353 fewer under templates with more shapes per layout and master.
354- **Parse-tree memory of an accepted part.** The guard's own lxml
355 walks hold every open element with its attributes, and its parses of
356 the skeleton parts (``_skeleton_xml``) the whole tree, about 200
357 bytes per attribute, as the loaders' own parses of the same part do.
358 ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` bounds one start tag, not how many
359 such tags are open at once: a slide, header or footer of nested
360 elements each carrying close to that many attributes costs about
361 14 MB per open level (eight levels, a 5.1 MB slide, held 118 MB in
362 the guard and 122 MB in python-pptx's load; a 128 MiB slide about
363 3 GB). ``document.xml`` and relationships parts are not affected: an
364 element over ``MAX_DOCX_ELEMENT_ATTRIBUTES`` or
365 ``MAX_RELS_ELEMENT_ATTRIBUTES`` attributes is refused as it starts.
366- **In-process parsers have no timeout.** python-docx, python-pptx,
367 openpyxl, pandas and unstructured run inside the indexing worker with
368 no wall clock, so super-linear behaviour other than the fan-outs
369 listed above can pin a worker; only the soffice and pandoc
370 subprocesses carry timeouts.
371"""
373from __future__ import annotations
375import functools
376import hashlib
377import io
378import posixpath
379import re
380import struct
381import zipfile
382import zlib
383from typing import Any, Callable, NamedTuple, Optional, cast
385from loguru import logger
387#: Extensions whose loaders parse zip containers of XML parts.
388#: (.doc/.ppt arrive as legacy OLE binaries and are converted to
389#: containers by the loader, so the guard does not apply to their
390#: raw upload bytes.)
391ZIP_CONTAINER_EXTENSIONS = frozenset(
392 {".docx", ".xlsx", ".odt", ".pptx", ".epub"}
393)
395#: Per-entry uncompressed ceiling. A text-heavy 200-page .docx keeps
396#: its main XML part in single-digit MB; an XML part near this ceiling
397#: already costs the in-process parsers on the order of 1-2+ GB of
398#: memory (XML parse trees run ~10-20x the XML size), so larger parts
399#: are refused rather than risk an OOM of the worker. This ceiling does
400#: not bound what a loader builds from the tree (for example pandas'
401#: .xlsx grid padding; see "Known gaps" in the module docstring).
402MAX_ENTRY_BYTES = 128 * 1024 * 1024
404#: Total uncompressed ceiling across all *parsed* (non-media) entries.
405#: Parsers that walk every part (openpyxl across worksheets,
406#: python-pptx across slides) pay the parse-tree multiplier on the
407#: total, so the total is kept to a few entry ceilings.
408MAX_TOTAL_BYTES = 512 * 1024 * 1024
410#: Per-entry ceiling for binary media entries (see module docstring).
411#: The packages that read media load each part once at ~1x, so a
412#: single large video or high-resolution image in a deck stays
413#: accepted, while a media "bomb" is still bounded.
414MAX_MEDIA_ENTRY_BYTES = 512 * 1024 * 1024
416#: Ceiling across *all* entries, media included. python-docx and
417#: python-pptx hold every part's blob at once, so this bounds the
418#: resident media (at ~1x) plus the parsed parts' raw bytes; it admits
419#: a media-heavy deck of several hundred MB. It also bounds the
420#: guard's own verification work (every DEFLATE entry is inflated
421#: once; a STORED entry's size is fixed by its equality check).
422MAX_CONTAINER_BYTES = 1024 * 1024 * 1024
424#: Entry-count ceiling: bounds central-directory walks and per-part
425#: parser work; real containers carry dozens of parts.
426MAX_ENTRIES = 10_000
429#: Names that are parsed as markup whatever their leading bytes say.
430_MARKUP_SUFFIXES = (
431 ".xml",
432 ".rels",
433 ".xhtml",
434 ".html",
435 ".htm",
436 ".opf",
437 ".ncx",
438 ".svg",
439)
441#: Leading-byte signatures of binary media no document parser reads as
442#: markup (none of them starts with ``<``).
443_MEDIA_SIGNATURES = (
444 b"\x89PNG\r\n\x1a\n", # PNG
445 b"\xff\xd8\xff", # JPEG
446 b"GIF87a",
447 b"GIF89a",
448 b"II*\x00", # TIFF little-endian
449 b"MM\x00*", # TIFF big-endian
450 b"BM", # BMP
451 b"RIFF", # WAV / AVI / WebP
452 b"\xd7\xcd\xc6\x9a", # placeable WMF
453 b"\x1f\x8b\x08", # gzip (EMZ/WMZ)
454 b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1", # OLE compound file
455 b"PK\x03\x04", # embedded package
456 b"%PDF-",
457 b"\x30\x26\xb2\x75\x8e\x66\xcf\x11", # ASF (WMV/WMA)
458 b"OggS",
459 b"fLaC",
460 b"ID3", # MP3
461 b"\x1a\x45\xdf\xa3", # Matroska / WebM
462 b"wOFF",
463 b"wOF2",
464 b"OTTO",
465 b"\x00\x01\x00\x00", # TrueType
466)
469class DecompressionBombError(ValueError):
470 """A zip-container document declares an unreasonable uncompressed size."""
473#: zipfile's local file header layout (``zipfile.structFileHeader``).
474_LOCAL_HEADER = struct.Struct("<4s2B4HL2L2H")
475_LOCAL_HEADER_MAGIC = b"PK\x03\x04"
476#: End-of-central-directory signature: the first bytes of an empty
477#: archive (which zipfile, and so the guard, accepts).
478_EMPTY_ARCHIVE_MAGIC = b"PK\x05\x06"
479#: A central-directory record is 46 fixed bytes plus its name and
480#: extra fields (both bounded by other checks once read); 1 KiB per
481#: allowed entry is a generous honest ceiling for the walk.
482_CENTRAL_DIR_BYTES_PER_ENTRY = 1024
483#: The function ``ZipFile._RealGetContents`` itself calls to pick the
484#: end records (``endrec = _EndRecData(fp)``), then walks
485#: ``endrec[_ECD_SIZE]`` bytes of central directory. Calling the
486#: running interpreter's own copy, rather than a re-implementation,
487#: is what makes the guard bound the very span zipfile then walks:
488#: the selection is subtle (the fixed 22-byte record at the end is
489#: taken before any backwards search, the ZIP64 record is followed
490#: through its locator's offset and may sit behind "extensible data",
491#: and that ZIP64 logic changed across 3.12 patch releases), and any
492#: divergence lets a writer show the guard one record and zipfile
493#: another. Private but present unchanged in name and return shape in
494#: every CPython the project supports (``requires-python`` < 3.15).
495_zipfile_end_record = zipfile._EndRecData # type: ignore[attr-defined]
496#: Index of the central-directory size in that record
497#: (``zipfile._ECD_SIZE``).
498_ECD_SIZE = 5
500#: General-purpose flag bits the local/central agreement check reads.
501_FLAG_DEFLATE_LEVEL_BITS = 0x0006 # compression-level hint, ignored
502_FLAG_DATA_DESCRIPTOR = 0x0008
503_FLAG_UTF8 = 0x0800
504#: Data-descriptor signature (optional before the descriptor fields).
505_DATA_DESCRIPTOR_MAGIC = b"PK\x07\x08"
506#: A 32-bit size of this value defers to the ZIP64 extra field.
507_ZIP64_LIMIT = 0xFFFFFFFF
509#: The only compression methods accepted: those the bounded inflation
510#: below can verify (zlib takes an output limit per step). zipfile
511#: decompresses BZIP2/LZMA/ZSTD chunks with no output limit at all, so
512#: a tiny entry of those methods can materialize gigabytes in any
513#: reader, this guard included. Office/ODF/EPUB writers only emit
514#: STORED and DEFLATE.
515_ALLOWED_METHODS = frozenset({zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED})
518def _max_compress_size(file_size: int) -> int:
519 """Largest compressed size accepted for a DEFLATE entry.
521 Incompressible data costs ~5 bytes per 64 KiB stored block plus
522 framing. A full read copies an entry's whole compressed range into
523 memory, so that range must not exceed the declared size by more
524 than this slack.
525 """
526 return file_size + (file_size >> 10) + 1024
529#: Inflation step sizes for the verification pass (memory stays at a
530#: few MiB whatever the entry size).
531_INFLATE_INPUT_CHUNK = 64 * 1024
532_INFLATE_OUTPUT_CHUNK = 1024 * 1024
535def _local_entry_range(
536 content: memoryview, info: zipfile.ZipInfo
537) -> tuple[int, int]:
538 """Byte range of *info*'s compressed data, after requiring the
539 entry's local header to agree with its central-directory record.
541 zipfile reads an entry through the central directory (offset,
542 method, sizes) and only takes the name/extra lengths from the
543 local header, but pandoc's zip reader takes the method and
544 compressed size from the *local* header. An entry whose two
545 headers disagree would therefore be verified here as one stream
546 and inflated by pandoc as another — for instance a small honest
547 central record over a local header declaring a much larger
548 stream. So the method, the general-purpose flags (bits 1-2, the
549 DEFLATE level hint, excepted), the CRC, both sizes and the name
550 must agree. With a data descriptor (flag bit 3), the local CRC and
551 sizes may each be zero (the usual streaming form) or the central
552 value, and the descriptor that follows the data must repeat the
553 central CRC and sizes exactly; the local data range is then the
554 central one by construction.
555 """
556 start = info.header_offset
557 header = bytes(content[start : start + _LOCAL_HEADER.size])
558 if len(header) != _LOCAL_HEADER.size: 558 ↛ 559line 558 didn't jump to line 559 because the condition on line 558 was never true
559 raise zipfile.BadZipFile("truncated local header")
560 (
561 magic,
562 _version,
563 _os,
564 flags,
565 method,
566 _time,
567 _date,
568 crc,
569 compress_size,
570 file_size,
571 name_len,
572 extra_len,
573 ) = _LOCAL_HEADER.unpack(header)
574 if magic != _LOCAL_HEADER_MAGIC: 574 ↛ 575line 574 didn't jump to line 575 because the condition on line 574 was never true
575 raise zipfile.BadZipFile("bad local header magic")
576 name_start = start + _LOCAL_HEADER.size
577 name = bytes(content[name_start : name_start + name_len])
578 extra = bytes(
579 content[name_start + name_len : name_start + name_len + extra_len]
580 )
581 if len(name) != name_len or len(extra) != extra_len: 581 ↛ 582line 581 didn't jump to line 582 because the condition on line 581 was never true
582 raise zipfile.BadZipFile("truncated local header")
584 zip64 = _ZIP64_LIMIT in (compress_size, file_size)
585 if zip64:
586 compress_size, file_size = _local_zip64_sizes(extra)
588 if method != info.compress_type:
589 raise DecompressionBombError(
590 "zip entry's local header disagrees with the central directory"
591 )
592 if (flags ^ info.flag_bits) & ~_FLAG_DEFLATE_LEVEL_BITS:
593 raise DecompressionBombError(
594 "zip entry's local header disagrees with the central directory"
595 )
596 encoding = "utf-8" if flags & _FLAG_UTF8 else "cp437"
597 if name != info.orig_filename.encode(encoding):
598 raise DecompressionBombError(
599 "zip entry's local header disagrees with the central directory"
600 )
601 declared = (crc, compress_size, file_size)
602 central = (info.CRC, info.compress_size, info.file_size)
603 if flags & _FLAG_DATA_DESCRIPTOR:
604 agrees = all(
605 local in (0, value) for local, value in zip(declared, central)
606 )
607 else:
608 agrees = declared == central
609 if not agrees:
610 raise DecompressionBombError(
611 "zip entry's local header disagrees with the central directory"
612 )
614 data_start = name_start + name_len + extra_len
615 data_end = data_start + info.compress_size
616 if data_end > len(content): 616 ↛ 617line 616 didn't jump to line 617 because the condition on line 616 was never true
617 raise zipfile.BadZipFile("entry data runs past the archive")
618 if flags & _FLAG_DATA_DESCRIPTOR:
619 _check_data_descriptor(content, data_end, info, zip64=zip64)
620 return data_start, data_end
623def _local_zip64_sizes(extra: bytes) -> tuple[int, int]:
624 """(compressed, uncompressed) sizes from a local ZIP64 extra field.
626 A local header that sets either 32-bit size to 0xFFFFFFFF must
627 carry both 64-bit sizes (uncompressed first) in its ZIP64 field.
628 """
629 pos = 0
630 while pos + 4 <= len(extra): 630 ↛ 639line 630 didn't jump to line 639 because the condition on line 630 was always true
631 tag, size = struct.unpack_from("<2H", extra, pos)
632 body = extra[pos + 4 : pos + 4 + size]
633 if tag == 0x0001: 633 ↛ 638line 633 didn't jump to line 638 because the condition on line 633 was always true
634 if len(body) < 16: 634 ↛ 635line 634 didn't jump to line 635 because the condition on line 634 was never true
635 break
636 file_size, compress_size = struct.unpack_from("<2Q", body)
637 return compress_size, file_size
638 pos += 4 + size
639 raise zipfile.BadZipFile("local header lacks its ZIP64 sizes")
642def _check_data_descriptor(
643 content: memoryview, offset: int, info: zipfile.ZipInfo, *, zip64: bool
644) -> None:
645 """Require the data descriptor at *offset* to repeat the central
646 CRC and sizes (signature optional; 8-byte sizes after a ZIP64
647 local header, 4-byte otherwise)."""
648 size_format = "<L2Q" if zip64 else "<3L"
649 record = struct.Struct(size_format)
650 tail = bytes(content[offset : offset + 4 + record.size])
651 candidates = [tail[: record.size]]
652 if tail[:4] == _DATA_DESCRIPTOR_MAGIC: 652 ↛ 654line 652 didn't jump to line 654 because the condition on line 652 was always true
653 candidates.append(tail[4:])
654 expected = (info.CRC, info.compress_size, info.file_size)
655 if not any(
656 len(fields) == record.size and record.unpack(fields) == expected
657 for fields in candidates
658 ):
659 raise DecompressionBombError(
660 "zip entry's data descriptor disagrees with the central directory"
661 )
664def _inflate_head(content: memoryview, info: zipfile.ZipInfo) -> bytes:
665 """Verify *info*'s data is exactly what it declares, and return its
666 first 64 bytes.
668 CPython's zipfile truncates an entry to its declared size only
669 *after* inflating each read chunk, and a full ``read()`` inflates
670 the entry's whole compressed range in one step, so a DEFLATE entry
671 that declares 100 bytes can still materialize gigabytes inside a
672 parser. Inflating here in bounded steps, and refusing any entry
673 whose real output exceeds its declaration, is what makes the
674 declared sizes an actual bound on parser reads.
676 The stream must also *end* exactly where it is declared to: the
677 inflater has to reach the end of the DEFLATE stream at the last
678 byte of the compressed range (no truncation, no trailing bytes),
679 having produced exactly the declared size with the declared CRC.
680 A reader that does not stop at the central directory's compressed
681 size (pandoc's, which follows the local header) would otherwise
682 keep inflating past the range this check saw.
683 """
684 data_start, data_end = _local_entry_range(content, info)
685 if info.compress_type == zipfile.ZIP_STORED:
686 crc = 0
687 for pos in range(data_start, data_end, _INFLATE_OUTPUT_CHUNK):
688 crc = zlib.crc32(
689 content[pos : min(pos + _INFLATE_OUTPUT_CHUNK, data_end)], crc
690 )
691 if crc != info.CRC: 691 ↛ 692line 691 didn't jump to line 692 because the condition on line 691 was never true
692 raise DecompressionBombError("zip entry fails its CRC check")
693 return bytes(content[data_start : min(data_end, data_start + 64)])
695 if data_start == data_end and info.file_size == 0 and info.CRC == 0: 695 ↛ 696line 695 didn't jump to line 696 because the condition on line 695 was never true
696 return b"" # an empty DEFLATE entry some writers emit bare
698 inflater = zlib.decompressobj(-zlib.MAX_WBITS)
699 produced = 0
700 crc = 0
701 head = b""
702 pos = data_start
703 pending = b""
704 while not inflater.eof:
705 if not pending and pos < data_end:
706 pending = bytes(
707 content[pos : min(pos + _INFLATE_INPUT_CHUNK, data_end)]
708 )
709 pos += len(pending)
710 # Once the range is consumed, ``pending`` is b"" and this drains
711 # output zlib still holds: a step that hit its output cap can
712 # consume the last input byte before emitting all its output
713 # (and before reaching the end of the stream).
714 out = inflater.decompress(pending, _INFLATE_OUTPUT_CHUNK)
715 pending = inflater.unconsumed_tail
716 if not out and not pending and pos >= data_end and not inflater.eof:
717 break # range consumed and drained before the stream ended
718 produced += len(out)
719 if produced > info.file_size:
720 raise DecompressionBombError(
721 "zip entry inflates past its declared size"
722 )
723 crc = zlib.crc32(out, crc)
724 if len(head) < 64:
725 head += out[: 64 - len(head)]
726 if not inflater.eof:
727 raise DecompressionBombError("zip entry's DEFLATE stream is truncated")
728 if pending or inflater.unused_data or pos != data_end:
729 raise DecompressionBombError(
730 "zip entry's DEFLATE stream ends before its compressed range"
731 )
732 if produced != info.file_size:
733 raise DecompressionBombError(
734 "zip entry inflates to less than its declared size"
735 )
736 if crc != info.CRC:
737 raise DecompressionBombError("zip entry fails its CRC check")
738 return head
741#: Containers whose entries may earn the media ceiling. Their parsers
742#: (python-docx/-pptx over lxml, openpyxl over lxml/expat) never parse
743#: a signature-headed media part as XML. EPUB/ODT go to pandoc, whose
744#: readers may still parse such an entry (an EPUB chapter is read by
745#: its manifest media type, not its bytes), so every EPUB/ODT entry
746#: stays under the parsed-part ceilings.
747MEDIA_CEILING_EXTENSIONS = frozenset({".docx", ".pptx", ".xlsx"})
750def _is_canonical_member_name(name: str) -> bool:
751 """True for a plain relative member name.
753 Refuses empty, ``.`` and ``..`` segments (so ``//``, ``/./``, a
754 leading ``/`` and traversal) and backslashes. Such names make
755 distinct members that readers normalise onto one path — openpyxl's
756 ``get_rels_path`` collapses ``a//b.xml`` and ``a/b.xml`` onto one
757 ``_rels`` part, for instance — and no Office/ODF/EPUB writer emits
758 them. A single trailing ``/`` (a directory entry) is allowed. NUL
759 never reaches here: zipfile truncates names at NUL when it reads
760 the central directory, and names that then collide are refused by
761 the duplicate-name check.
762 """
763 if not name or "\\" in name:
764 return False
765 body = name[:-1] if name.endswith("/") else name
766 return all(segment not in ("", ".", "..") for segment in body.split("/"))
769def _is_media_head(info: zipfile.ZipInfo, head: bytes) -> bool:
770 """True when *info* is binary media by content (see module docstring)."""
771 if info.is_dir(): 771 ↛ 772line 771 didn't jump to line 772 because the condition on line 771 was never true
772 return False
773 if info.filename.lower().endswith(_MARKUP_SUFFIXES):
774 return False
775 # ISO BMFF (MP4 / MOV / M4A / HEIC): a big-endian box size of at
776 # least 8 (so the head starts with two zero bytes) then "ftyp".
777 # Checked before the markup sniff: such a head cannot parse as XML
778 # in any encoding ("ftyp" read as UCS-4 is above U+10FFFF, and two
779 # leading zero bytes are a NUL in UTF-16), while a real MP4 whose
780 # ftyp box is 60 bytes long starts 00 00 00 3C, which the UTF-32BE
781 # decode below would otherwise read as "<".
782 if (
783 head[:2] == b"\x00\x00"
784 and head[4:8] == b"ftyp"
785 and int.from_bytes(head[:4], "big") >= 8
786 ):
787 return True
788 if _looks_like_markup(head):
789 return False
790 if head.startswith(_MEDIA_SIGNATURES):
791 return True
792 # EMF: EMR_HEADER record type 1 with the " EMF" signature at 40.
793 return head[:4] == b"\x01\x00\x00\x00" and head[40:44] == b" EMF"
796#: Encodings an XML parser can auto-detect a document start in.
797_MARKUP_ENCODINGS = (
798 "utf-8",
799 "utf-16-le",
800 "utf-16-be",
801 "utf-32-le",
802 "utf-32-be",
803)
806def _looks_like_markup(head: bytes) -> bool:
807 """True when *head* could begin an XML document.
809 Conservative: a match only ever keeps an entry under the parsed-part
810 ceilings. Undecodable bytes become U+FFFD, never ``<``.
811 """
812 for encoding in _MARKUP_ENCODINGS:
813 text = head.decode(encoding, errors="replace")
814 if text.lstrip("\ufeff \t\r\n").startswith("<"):
815 return True
816 return False
819#: unstructured's ``_ZipFileDetector`` rules, in its order: the first
820#: member-name pattern any member matches (``re.match``) names the
821#: package type; with no match it reads a root ``mimetype`` member as
822#: the type.
823_SNIFFED_ZIP_TYPES = (
824 (".docx", re.compile(r"word/document.*\.xml$")),
825 (".xlsx", re.compile(r"xl/workbook.*\.xml$")),
826 (".pptx", re.compile(r"ppt/presentation.*\.xml$")),
827)
829#: The member pandas' ``inspect_excel_format`` needs (names lowercased)
830#: to read a workbook with openpyxl rather than another engine.
831_PANDAS_XLSX_MEMBER = "xl/workbook.xml"
833#: The root ``mimetype`` member of an ODF or EPUB package, and the media
834#: type it must name for each extension when present.
835_MIMETYPE_MEMBER = "mimetype"
836_PACKAGE_MIMETYPES = {
837 ".odt": "application/vnd.oasis.opendocument.text",
838 ".epub": "application/epub+zip",
839}
841#: Longest ``mimetype`` member read (the media types above are at most
842#: 39 bytes).
843_MAX_MIMETYPE_MEMBER_BYTES = 256
846def _check_package_type(archive: zipfile.ZipFile, ext: str) -> None:
847 """Refuse a package whose member layout makes a content sniffer pick
848 a different parser than its extension does.
850 langchain's Word and PowerPoint loaders call unstructured's
851 ``detect_filetype`` (whenever python-magic imports) and route a file
852 it calls DOC or PPT to ``partition_doc``/``partition_ppt``, which run
853 soffice with no timeout and partition its output without this
854 guard. For a zip the detector returns DOCX, XLSX or PPTX from the
855 first of its member-name patterns that matches
856 (``_SNIFFED_ZIP_TYPES``), and otherwise trusts a root ``mimetype``
857 member's content: a ``.docx`` with no ``word/document*.xml`` and a
858 ``mimetype`` of ``application/msword`` went to soffice. An OOXML
859 package (``.docx``/``.xlsx``/``.pptx``) is therefore refused when it
860 has a ``mimetype`` member (OPC packages have none) or when the
861 detector's patterns would not name its own extension, and an
862 ``.xlsx`` also when pandas would not pick openpyxl for it (no
863 ``xl/workbook.xml``; pandas would pick pyxlsb or odfpy instead). An
864 ``.odt`` or ``.epub`` (converted by pandoc with an explicit input
865 format, never sniffed on the upload path) is refused when its
866 ``mimetype`` member, if present, names another type. Called after
867 the entries are verified, so the ``mimetype`` read is bounded."""
868 names = archive.namelist()
869 if ext in _PACKAGE_MIMETYPES:
870 if _MIMETYPE_MEMBER not in names:
871 return
872 info = archive.getinfo(_MIMETYPE_MEMBER)
873 if info.file_size > _MAX_MIMETYPE_MEMBER_BYTES:
874 raise DecompressionBombError("package mimetype member is too long")
875 try:
876 declared: Optional[str] = archive.read(info).decode("utf-8").strip()
877 except UnicodeDecodeError:
878 declared = None
879 if declared != _PACKAGE_MIMETYPES[ext]:
880 raise DecompressionBombError(
881 "package mimetype does not match its extension"
882 )
883 return
884 if _MIMETYPE_MEMBER in names:
885 raise DecompressionBombError(
886 "office package has a mimetype member, which a content sniffer "
887 "reads as its type"
888 )
889 sniffed = next(
890 (
891 kind
892 for kind, pattern in _SNIFFED_ZIP_TYPES
893 if any(pattern.match(name) for name in names)
894 ),
895 None,
896 )
897 if sniffed != ext:
898 raise DecompressionBombError(
899 "office package's part names do not match its extension"
900 )
901 if ext == ".xlsx" and _PANDAS_XLSX_MEMBER not in {
902 name.lower() for name in names
903 }:
904 raise DecompressionBombError(
905 "spreadsheet package has no xl/workbook.xml part"
906 )
909def _declared_central_directory_size(content: bytes) -> int:
910 """Read the central-directory size zipfile will walk, before
911 ``zipfile`` is asked to walk the records.
913 The end records are selected by zipfile's own ``_EndRecData`` — the
914 same call ``ZipFile`` makes — so the size returned is exactly the
915 one zipfile then reads (``endrec[_ECD_SIZE]``, which the ZIP64
916 record overrides whenever zipfile accepts one). It reads only the
917 end records and walks none. Fails closed: when zipfile would find
918 no usable end record, or rejects the one it finds, the container
919 is refused here rather than the bound standing aside.
920 """
921 try:
922 endrec = _zipfile_end_record(io.BytesIO(content))
923 except (zipfile.BadZipFile, OSError, struct.error) as exc:
924 raise DecompressionBombError("malformed zip container") from exc
925 if not endrec:
926 raise DecompressionBombError("malformed zip container")
927 return int(endrec[_ECD_SIZE])
930def _check_central_directory_span(content: bytes, *, max_entries: int) -> None:
931 """Refuse a central directory whose declared byte span alone
932 exceeds the entry-count-scaled bound.
934 ``zipfile`` materialises every central-directory record before the
935 entry-count ceiling can count them (a hand-packed record flood a
936 fraction of the upload cap in size buys hundreds of MiB of RSS),
937 and the EOCD's record count wraps at 65,535 — zipfile ignores it —
938 so only the declared byte span bounds that walk up front. A
939 container whose end records zipfile cannot read is refused too.
940 """
941 cd_size = _declared_central_directory_size(content)
942 limit = max_entries * _CENTRAL_DIR_BYTES_PER_ENTRY
943 if cd_size > limit:
944 raise DecompressionBombError(
945 f"zip container's central directory spans {cd_size} bytes, "
946 f"exceeding the {limit}-byte guard limit"
947 )
950def validate_zip_container(
951 content: bytes,
952 extension: str,
953 *,
954 max_entry_bytes: int = MAX_ENTRY_BYTES,
955 max_total_bytes: int = MAX_TOTAL_BYTES,
956 max_media_entry_bytes: int = MAX_MEDIA_ENTRY_BYTES,
957 max_container_bytes: int = MAX_CONTAINER_BYTES,
958 max_entries: int = MAX_ENTRIES,
959 zip_extensions: frozenset[str] = ZIP_CONTAINER_EXTENSIONS,
960) -> None:
961 """Refuse zip-container documents whose declared expansion is hostile.
963 Raises ``DecompressionBombError`` (a ``ValueError``) when the
964 container declares a central-directory byte span over
965 ``max_entries`` kibibytes (checked before ``zipfile`` materialises
966 the records); declares more entries than ``max_entries``; a parsed
967 part over ``max_entry_bytes`` or parsed parts over
968 ``max_total_bytes`` in total; a binary media entry over
969 ``max_media_entry_bytes``; more than ``max_container_bytes``
970 across all entries; or when it is not a readable zip at all; or
971 when its layout lets a content sniffer pick another parser than
972 the extension does (``_check_package_type``). Non-container
973 extensions are returned untouched, and rejection happens before
974 any parser sees the bytes.
975 """
976 ext = (
977 extension.lower()
978 if extension.startswith(".")
979 else f".{extension.lower()}"
980 )
981 if ext not in zip_extensions:
982 return
984 # zipfile finds the archive from the end of the file, so anything
985 # may precede it; content sniffers (unstructured) read the start.
986 if not content.startswith((_LOCAL_HEADER_MAGIC, _EMPTY_ARCHIVE_MAGIC)):
987 raise DecompressionBombError(
988 "zip container does not start with a zip signature"
989 )
991 # zipfile materialises every central-directory record before the
992 # entry-count ceiling can refuse them, so the span the end records
993 # declare is bounded first — whatever their (wrapped, ignored)
994 # record counts claim.
995 _check_central_directory_span(content, max_entries=max_entries)
997 try:
998 with zipfile.ZipFile(io.BytesIO(content)) as archive:
999 parsed_total, container_total, entries = _check_entries(
1000 archive,
1001 memoryview(content),
1002 max_entry_bytes=max_entry_bytes,
1003 max_total_bytes=max_total_bytes,
1004 max_media_entry_bytes=max_media_entry_bytes,
1005 max_container_bytes=max_container_bytes,
1006 max_entries=max_entries,
1007 media_allowed=ext in MEDIA_CEILING_EXTENSIONS,
1008 )
1009 _check_package_type(archive, ext)
1010 if ext == ".xlsx":
1011 _check_xlsx_sheet_fan_out(content)
1012 elif ext == ".pptx":
1013 _check_pptx_slide_fan_out(content)
1014 elif ext == ".docx":
1015 _check_docx_section_fan_out(content)
1016 except DecompressionBombError:
1017 raise
1018 except (
1019 zipfile.BadZipFile,
1020 # A container the stdlib cannot even list or open (undecodable
1021 # UTF-8-flagged names, unsupported versions or compression,
1022 # encrypted entries, truncated records, corrupt deflate data)
1023 # is refused too: fail closed, never pass it on.
1024 ValueError,
1025 NotImplementedError,
1026 RuntimeError,
1027 EOFError,
1028 OSError,
1029 struct.error,
1030 zlib.error,
1031 ) as exc:
1032 raise DecompressionBombError("malformed zip container") from exc
1034 # Deliberately logs no user-controlled content: entry count and
1035 # declared sizes only.
1036 logger.debug(
1037 "zip container accepted: {} entries, {} parsed / {} total "
1038 "declared bytes",
1039 entries,
1040 parsed_total,
1041 container_total,
1042 )
1045def _check_entries(
1046 archive: zipfile.ZipFile,
1047 content: memoryview,
1048 *,
1049 max_entry_bytes: int,
1050 max_total_bytes: int,
1051 max_media_entry_bytes: int,
1052 max_container_bytes: int,
1053 max_entries: int,
1054 media_allowed: bool,
1055) -> tuple[int, int, int]:
1056 infos = archive.infolist()
1057 if len(infos) > max_entries:
1058 raise DecompressionBombError(
1059 f"zip container declares {len(infos)} entries "
1060 f"(ceiling {max_entries})"
1061 )
1063 # Data in front of the archive shifts every header offset (zipfile
1064 # compensates); a genuine container's first entry starts at 0.
1065 if infos and min(info.header_offset for info in infos) != 0:
1066 raise DecompressionBombError(
1067 "zip container has data before its first entry"
1068 )
1070 # Pass 1: declaration-only checks, before any entry data is touched.
1071 largest_entry_ceiling = max(max_entry_bytes, max_media_entry_bytes)
1072 header_offsets: set[int] = set()
1073 names: set[str] = set()
1074 container_total = 0
1075 for info in infos:
1076 if not _is_canonical_member_name(info.filename):
1077 raise DecompressionBombError(
1078 "zip container has a non-canonical member name"
1079 )
1080 if info.filename in names:
1081 raise DecompressionBombError(
1082 "zip container has two members with the same name"
1083 )
1084 names.add(info.filename)
1085 if info.header_offset in header_offsets:
1086 raise DecompressionBombError(
1087 "zip container has entries sharing one local header"
1088 )
1089 header_offsets.add(info.header_offset)
1090 if info.file_size > largest_entry_ceiling:
1091 raise DecompressionBombError(
1092 f"zip entry declares {info.file_size} bytes "
1093 f"(ceiling {largest_entry_ceiling})"
1094 )
1095 if info.compress_type not in _ALLOWED_METHODS:
1096 raise DecompressionBombError(
1097 "zip container uses an unsupported compression method"
1098 )
1099 if info.flag_bits & 0x1:
1100 raise DecompressionBombError("zip container has encrypted entries")
1101 if info.compress_size > _max_compress_size(info.file_size) or (
1102 info.compress_type == zipfile.ZIP_STORED
1103 and info.compress_size != info.file_size
1104 ):
1105 raise DecompressionBombError(
1106 "zip entry's compressed size does not match its declared size"
1107 )
1108 container_total += info.file_size
1109 if container_total > max_container_bytes:
1110 raise DecompressionBombError(
1111 f"zip container declares over {max_container_bytes} bytes "
1112 "total uncompressed"
1113 )
1115 # Pass 2: verify each declaration by bounded inflation (total work
1116 # bounded by pass 1's container total), then classify and apply the
1117 # per-kind ceilings.
1118 parsed_total = 0
1119 for info in infos:
1120 head = _inflate_head(content, info)
1121 if media_allowed and _is_media_head(info, head):
1122 # Binary media is held to its own per-entry ceiling and
1123 # counts only toward the whole-container total.
1124 if info.file_size > max_media_entry_bytes: 1124 ↛ 1125line 1124 didn't jump to line 1125 because the condition on line 1124 was never true
1125 raise DecompressionBombError(
1126 f"zip media entry {info.filename!r} declares "
1127 f"{info.file_size} bytes "
1128 f"(ceiling {max_media_entry_bytes})"
1129 )
1130 continue
1131 if info.file_size > max_entry_bytes:
1132 raise DecompressionBombError(
1133 f"zip entry {info.filename!r} declares "
1134 f"{info.file_size} bytes (ceiling {max_entry_bytes})"
1135 )
1136 parsed_total += info.file_size
1137 if parsed_total > max_total_bytes:
1138 raise DecompressionBombError(
1139 f"zip container declares over {max_total_bytes} bytes "
1140 "of parsed parts uncompressed"
1141 )
1142 return parsed_total, container_total, len(infos)
1145#: The parser families openpyxl's read path (and msoffcrypto, which
1146#: unstructured runs first) parses ``.xlsx`` parts with: lxml
1147#: (``openpyxl.xml.functions.fromstring``) for the package skeleton, the
1148#: styles, the document properties, chartsheets and every relationships
1149#: part, and expat (``xml.etree``/defusedxml ``iterparse``) for
1150#: worksheets and shared strings; msoffcrypto reads
1151#: ``[Content_Types].xml`` with minidom, expat too.
1152_LXML = "lxml"
1153_EXPAT = "expat"
1156class _XlsxScanTarget:
1157 """What one parser of ``_scan_xlsx_part`` reports to: an lxml parser
1158 target, or the handlers of an expat parser (``expat_start``), whose
1159 element and attribute names pyexpat interns in *interned*, so they
1160 are counted from there, after each chunk. It refuses a namespace URI
1161 over ``MAX_XLSX_NAMESPACE_URI_CHARS`` characters, a DTD, nesting
1162 deeper than ``MAX_XLSX_XML_DEPTH`` and more than
1163 ``MAX_XLSX_PART_NAMES`` distinct names, and builds nothing."""
1165 def __init__(self, interned: Optional[dict] = None):
1166 self.depth = 0
1167 self.names: set[str] = set()
1168 self.interned = interned
1170 def start_ns(self, prefix: Optional[str], uri: Optional[str]) -> None:
1171 if uri is not None and len(uri) > MAX_XLSX_NAMESPACE_URI_CHARS:
1172 raise DecompressionBombError(
1173 "spreadsheet part declares a namespace URI of more "
1174 f"than {MAX_XLSX_NAMESPACE_URI_CHARS} characters"
1175 )
1176 self.names.add(f"xmlns:{prefix or ''}")
1177 self._count_names()
1179 def doctype(self, *_declaration) -> None:
1180 raise DecompressionBombError("document package part declares a DTD")
1182 def start(self, tag: str, attributes: dict) -> None:
1183 self.names.add(tag)
1184 if attributes:
1185 self.names.update(attributes)
1186 self._count_names()
1187 self.expat_start(tag, attributes)
1189 def expat_start(self, _tag: str, _attributes) -> None:
1190 self.depth += 1
1191 if self.depth > MAX_XLSX_XML_DEPTH:
1192 raise DecompressionBombError(
1193 "spreadsheet part nests elements more than "
1194 f"{MAX_XLSX_XML_DEPTH} deep"
1195 )
1197 def _count_names(self) -> None:
1198 interned = len(self.interned) if self.interned is not None else 0
1199 if len(self.names) + interned > MAX_XLSX_PART_NAMES:
1200 raise DecompressionBombError(
1201 "spreadsheet part uses more than "
1202 f"{MAX_XLSX_PART_NAMES} distinct names"
1203 )
1205 def end(self, _tag) -> None:
1206 self.depth -= 1
1208 def close(self) -> None:
1209 return None
1212def _check_xlsx_namespaces(
1213 archive: zipfile.ZipFile,
1214 parts: dict[str, frozenset[str]],
1215 scanned: set[tuple[str, str]],
1216) -> None:
1217 """Stream each of *parts* (member name -> parser families) once,
1218 with the parsers of the families the loaders parse it with, before
1219 any of them parses it (pairs in *scanned* are skipped and new ones
1220 added). A missing member is skipped: openpyxl skips it too or fails
1221 to load.
1223 Refused, in every member scanned: an encoding other than UTF-8
1224 (``_require_utf8_xml``), a start tag that may carry more than
1225 ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` attributes (``_XmlTagSigns``) and a
1226 DTD (``_XmlProlog``; see ``_refuse_dtd``: it could spell a
1227 declaration through an entity), all checked before either parser
1228 sees the bytes (``_CountedXmlStream``; the lxml push parser buffered
1229 a whole internal subset before its doctype callback ran); then, as
1230 ``_XlsxScanTarget`` receives them, a namespace URI over
1231 ``MAX_XLSX_NAMESPACE_URI_CHARS`` characters (lxml and expat copy it
1232 into the name of every element under it), a DTD again (a second
1233 line), nesting deeper than ``MAX_XLSX_XML_DEPTH`` and more than
1234 ``MAX_XLSX_PART_NAMES`` distinct names: both parsers keep state for
1235 every open element (expat about 145 bytes) and every distinct name
1236 for the whole parse, and neither, as the scan drives it, sets a
1237 limit. No tree is built. An lxml member is fed to an lxml parser
1238 with the options of openpyxl's parser and is refused when that
1239 parse fails: openpyxl's load of a part libxml2 rejects fails too,
1240 and its parse from a bytes buffer decodes UTF-32 with a byte-order
1241 mark, which this push parser rejects at the first byte (the
1242 encoding check before it refuses it first). An expat member is fed to an expat parser
1243 with namespace processing, as ``xml.etree`` creates it, until its
1244 first syntax error, where the loader's parse ends too. Python runs
1245 once per chunk, per declaration and per element start and end."""
1246 for name, families in parts.items():
1247 todo = frozenset(f for f in families if (name, f) not in scanned)
1248 if not todo: 1248 ↛ 1249line 1248 didn't jump to line 1249 because the condition on line 1248 was never true
1249 continue
1250 scanned.update((name, family) for family in todo)
1251 try:
1252 info = archive.getinfo(name)
1253 except KeyError:
1254 continue
1255 _scan_xlsx_part(archive, info, todo)
1258def _scan_xlsx_part(
1259 archive: zipfile.ZipFile, info: zipfile.ZipInfo, families: frozenset[str]
1260) -> None:
1261 """Scan one member for ``_check_xlsx_namespaces``."""
1262 from lxml import etree
1264 # nosemgrep: python.lang.security.use-defused-xml.use-defused-xml, reason: bounded OOXML scan refuses DTD and entity declarations
1265 from xml.parsers import expat
1267 # Each live parser: (feed, finish, the errors that end its parse,
1268 # whether such an error refuses the member).
1269 live: list[
1270 tuple[
1271 Callable[..., Any],
1272 Callable[..., Any],
1273 tuple[type[Exception], ...],
1274 bool,
1275 ]
1276 ] = []
1277 if _LXML in families:
1278 # openpyxl's parser: XMLParser(resolve_entities=False), whose
1279 # other options (no network, no huge tree) are lxml's defaults.
1280 lxml_parser = etree.XMLParser(
1281 target=_XlsxScanTarget(), resolve_entities=False, no_network=True
1282 )
1283 live.append(
1284 (lxml_parser.feed, lxml_parser.close, (etree.XMLSyntaxError,), True)
1285 )
1286 if _EXPAT in families:
1287 # The separator xml.etree uses; any turns namespace processing
1288 # on, which decides what expat rejects (an unbound prefix).
1289 # pyexpat interns every element and attribute name it passes
1290 # the handlers in this dict (as expat keeps its own entry for
1291 # each), so its size is the distinct names read so far.
1292 interned: dict = {}
1293 expat_parser = expat.ParserCreate(
1294 namespace_separator="}", intern=interned
1295 )
1296 expat_target = _XlsxScanTarget(interned)
1297 expat_parser.StartNamespaceDeclHandler = expat_target.start_ns
1298 expat_parser.StartDoctypeDeclHandler = expat_target.doctype
1299 expat_parser.ordered_attributes = True # a list costs less
1300 expat_parser.StartElementHandler = expat_target.expat_start
1301 expat_parser.EndElementHandler = expat_target.end
1303 # pyexpat reports an encoding it cannot decode as a LookupError
1304 # or ValueError, where xml.etree's parse fails too.
1305 def expat_feed(chunk: bytes, final: bool = False) -> None:
1306 try:
1307 expat_parser.Parse(chunk, final)
1308 finally:
1309 # At most one chunk's names past the ceiling.
1310 expat_target._count_names()
1312 live.append(
1313 (
1314 expat_feed,
1315 functools.partial(expat_feed, b"", True),
1316 (expat.ExpatError, LookupError, ValueError),
1317 False,
1318 )
1319 )
1320 with archive.open(info) as member:
1321 stream = _CountedXmlStream(member)
1322 while live and (chunk := stream.read(_XML_SCAN_CHUNK)):
1323 for parser in list(live):
1324 feed, _finish, errors, refuse = parser
1325 if not _xlsx_scan_step( 1325 ↛ 1328line 1325 didn't jump to line 1328 because the condition on line 1325 was never true
1326 functools.partial(feed, chunk), errors, refuse
1327 ):
1328 live.remove(parser)
1329 for _feed, finish, errors, refuse in live:
1330 _xlsx_scan_step(finish, errors, refuse)
1333def _xlsx_scan_step(step, errors: tuple, refuse: bool) -> bool:
1334 """Run ``step()`` for ``_scan_xlsx_part``; False when it raises one
1335 of *errors*, where that parser's parse of the member ends, or a
1336 refusal when *refuse* is set. A refusal (a ``ValueError`` too)
1337 propagates."""
1338 try:
1339 step()
1340 except DecompressionBombError:
1341 raise
1342 except errors as exc:
1343 if refuse: 1343 ↛ 1348line 1343 didn't jump to line 1348 because the condition on line 1343 was always true
1344 raise DecompressionBombError(
1345 "malformed spreadsheet package: a part openpyxl parses "
1346 "with lxml is not XML lxml reads"
1347 ) from exc
1348 return False
1349 return True
1352#: Chunk size of the namespace scan and of ``_CountedXmlStream``; at
1353#: most ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` bytes (see ``_XmlTagSigns``).
1354_XML_SCAN_CHUNK = 64 * 1024
1356#: Ceiling on the element nesting depth of a scanned ``.xlsx`` part.
1357#: expat, which openpyxl streams worksheets and shared strings with
1358#: (and msoffcrypto's minidom ``[Content_Types].xml``), keeps about 145
1359#: bytes per open element and sets no limit, so 128 MiB of ``<a>``, a
1360#: 130 KB file, would hold about 6 GB in expat alone (and
1361#: xml.etree builds a tree element per level); libxml2's push parser
1362#: driving a target, as the scan drives it, keeps about 36 bytes per
1363#: level and sets no limit either. libxml2 building a tree, as
1364#: openpyxl's parse of the other parts does, refuses more than 256
1365#: levels; this is the same ceiling. Office writers nest worksheet
1366#: elements about ten deep.
1367MAX_XLSX_XML_DEPTH = 256
1369#: Ceiling on the distinct names (element names, attribute names and
1370#: declared namespace prefixes, an element or attribute name counted
1371#: with its namespace URI) in a scanned ``.xlsx`` part. expat keeps an
1372#: entry for every distinct element and attribute name and every
1373#: prefix for the whole parse (about 140 bytes per element name), as
1374#: libxml2 keeps its name dictionary, so 1,000,000 distinct element
1375#: names, 8.9 MB of a worksheet, held about 140 MB in the scan (a
1376#: 128 MiB part about 2 GB). SpreadsheetML, its extensions and the
1377#: package parts define a few hundred names.
1378MAX_XLSX_PART_NAMES = 10_000
1380#: Ceiling on the ``=`` characters between two ``<`` in an XML part
1381#: the guard streams or counts (every ``.xlsx`` part the namespace scan
1382#: covers, and each ``.docx``/``.pptx`` part and relationships part the
1383#: guard parses): an upper bound on the attributes of a start tag,
1384#: since an attribute value cannot hold a ``<``. It is counted in the
1385#: part's bytes, which ``_require_utf8_xml`` holds to UTF-8 (no
1386#: multi-byte UTF-8 sequence holds either byte), before any parser
1387#: receives them. An attribute without ``=`` is a syntax error, and the
1388#: only other source of attributes, the defaults of a DTD's
1389#: attribute-list declarations (``xmlns:`` ones included), is refused
1390#: in the same bytes before then (``_XmlProlog``), so no parser builds
1391#: a start tag of more attributes.
1392#: expat keeps about 108 bytes per attribute of the tag it is parsing
1393#: and sets no limit (one tag of 1,000,000 attributes, 11 MB, held
1394#: about 175 MB in the scan). libxml2 refuses a start tag over 10 MB
1395#: up front only when it parses from a buffer (``fromstring``): its
1396#: push parser, as the scan drives it, buffers the whole tag and
1397#: reports that limit only once the tag is complete (4,000,000
1398#: attributes, 47 MB, held 551 MB first), and lxml's ``iterparse``,
1399#: which the ``.docx``/``.pptx`` walkers and the relationships check
1400#: use, builds such an element in full (2,000,000 attributes, 23 MB,
1401#: held 406 MB; one of 1,000,000 cost the guard about 200 MB in a
1402#: ``.docx``, ``.pptx`` or relationships part). Office writers put a
1403#: few dozen attributes on an element; this also admits a text node of
1404#: 32,767 ``=``, the most a cell holds in Excel.
1405MAX_XML_TAG_ATTRIBUTE_SIGNS = 65_536
1407#: An XML declaration as XML 1.0 (section 2.8) spells it, in bytes: the
1408#: version, then optionally the encoding name (group 1 or 2) and the
1409#: standalone flag, separated by XML whitespace.
1410_XML_DECLARATION = re.compile(
1411 rb"<\?xml[ \t\r\n]+version[ \t\r\n]*=[ \t\r\n]*"
1412 rb"(?:\"1\.[0-9]+\"|'1\.[0-9]+')"
1413 rb"(?:[ \t\r\n]+encoding[ \t\r\n]*=[ \t\r\n]*"
1414 rb"(?:\"([A-Za-z][A-Za-z0-9._-]*)\"|'([A-Za-z][A-Za-z0-9._-]*)'))?"
1415 rb"(?:[ \t\r\n]+standalone[ \t\r\n]*=[ \t\r\n]*"
1416 rb"(?:\"(?:yes|no)\"|'(?:yes|no)'))?"
1417 rb"[ \t\r\n]*\?>"
1418)
1421def _require_utf8_xml(head: bytes) -> None:
1422 """Refuse an XML part starting with *head* (its first
1423 ``_XML_SCAN_CHUNK`` bytes, or all of a shorter part) unless both
1424 parsers read it as UTF-8: an optional UTF-8 byte-order mark, then
1425 ``<`` or XML whitespace, no NUL byte anywhere in *head*, and either
1426 no XML declaration or one, complete within *head*, that names no
1427 encoding or names UTF-8 (in any case).
1429 The guard counts a part's ``<`` and ``=`` in bytes
1430 (``_XmlTagSigns``), which holds only for an ASCII-compatible
1431 encoding both parsers agree on, and they do not agree on UTF-16:
1432 expat reads a part whose first or second byte is zero as UTF-16
1433 without a byte-order mark (`` \\x00<\\x00`` as UTF-16LE), where
1434 libxml2 sees UTF-8 and fails, and libxml2 switches to UTF-16 after
1435 an ASCII declaration naming ``UTF-16LE``, which expat rejects. In
1436 UTF-16 a character whose low byte is ``3C`` (U+013C) is a ``<`` to
1437 a byte count, so a start tag of millions of attributes passed it.
1438 OPC (ECMA-376 Part 2) allows XML parts in UTF-8 or UTF-16, but
1439 every producer surveyed writes UTF-8 (Excel, Word and PowerPoint;
1440 LibreOffice's serializer, which writes nothing else; openpyxl,
1441 XlsxWriter, python-docx, python-pptx and SheetJS), as do all 2,238
1442 XML members of the 88 local Office files checked; the only UTF-16
1443 package found is a reader's test fixture (PhpSpreadsheet's). The NUL and
1444 first-byte tests also refuse UTF-32 and EBCDIC (``4C 6F A7 94``),
1445 which libxml2 detects and expat does not, and the declaration test
1446 every other declared encoding."""
1447 body = head[3:] if head.startswith(b"\xef\xbb\xbf") else head
1448 if b"\x00" in head or (body and body[:1] not in b"< \t\r\n"):
1449 raise DecompressionBombError(
1450 "document package part is not encoded in UTF-8"
1451 )
1452 if body.startswith(b"<?xml") and body[5:6] in (b" ", b"\t", b"\r", b"\n"):
1453 declaration = _XML_DECLARATION.match(body)
1454 if declaration is None:
1455 raise DecompressionBombError(
1456 "document package part's XML declaration is malformed "
1457 "or too long"
1458 )
1459 name = declaration.group(1) or declaration.group(2)
1460 if name is not None and name.lower() != b"utf-8":
1461 raise DecompressionBombError(
1462 "document package part is not encoded in UTF-8"
1463 )
1466class _XmlTagSigns:
1467 """Count the ``=`` between two ``<`` of a UTF-8 part
1468 (``_require_utf8_xml``) streamed in chunks of at most
1469 ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` bytes, and refuse more than that.
1470 Bytes are counted: no multi-byte UTF-8 sequence holds either byte.
1471 Only the run open at a chunk's end is carried: a run inside one
1472 chunk is shorter than the ceiling."""
1474 def __init__(self) -> None:
1475 self._open = 0
1477 def feed(self, chunk: bytes) -> None:
1478 first = chunk.find(b"<")
1479 if first < 0:
1480 self._open += chunk.count(b"=")
1481 else:
1482 self._open += chunk.count(b"=", 0, first)
1483 self._check()
1484 self._open = chunk.count(b"=", chunk.rfind(b"<"))
1485 self._check()
1487 def _check(self) -> None:
1488 if self._open > MAX_XML_TAG_ATTRIBUTE_SIGNS:
1489 raise DecompressionBombError(
1490 "document package part has a start tag that may carry "
1491 f"more than {MAX_XML_TAG_ATTRIBUTE_SIGNS} attributes"
1492 )
1495class _XmlProlog:
1496 """Refuse a markup declaration in the prolog of a UTF-8 part
1497 (``_require_utf8_xml``) streamed in chunks: any ``<!`` before the
1498 root element's start tag that does not open a comment (``<!--``).
1500 A document type declaration, the only place an ``<!ENTITY``,
1501 ``<!ATTLIST``, ``<!ELEMENT`` or ``<!NOTATION`` can stand, is legal
1502 only there; in content, or before the root with no ``<!DOCTYPE``,
1503 lxml and expat fail on ``<!`` that opens neither a comment nor a
1504 CDATA section without reading a declaration. Refusing it in the
1505 bytes keeps the internal subset from every parser: libxml2 reads
1506 (and its push parser first buffers) the whole subset before the
1507 root's first event, where ``_refuse_dtd`` looks, building every
1508 declaration (one million attribute declarations, a 17.9 MB part,
1509 held 424 MB), and attribute defaults it declares, ``xmlns:`` ones
1510 included, are added to the start tags they name, attributes the
1511 ``=`` count (``_XmlTagSigns``) never sees. Comments and processing
1512 instructions before the root are skipped as the parsers end them
1513 (``-->``, ``?>``); ``<!-`` without a second ``-``, or ``--``
1514 inside a comment, which both parsers reject, is refused.
1515 Scanning stops at the first ``<`` that opens neither, the root
1516 element (or markup both parsers reject), so ``<!`` in content,
1517 comments and CDATA sections is never examined."""
1519 def __init__(self) -> None:
1520 self._state = "prolog" # or "dash", "comment", "pi", "done"
1521 self._carry = b""
1523 def feed(self, chunk: bytes) -> None:
1524 if self._state == "done":
1525 return
1526 data = self._carry + chunk
1527 self._carry = b""
1528 at = 0
1529 while True:
1530 if self._state == "pi":
1531 end = data.find(b"?>", at)
1532 if end < 0:
1533 if data.endswith(b"?") and len(data) - 1 >= at:
1534 self._carry = b"?"
1535 return
1536 at = end + 2
1537 self._state = "prolog"
1538 elif self._state == "dash": # after "<!-"
1539 if at >= len(data):
1540 return
1541 if data[at : at + 1] != b"-":
1542 raise DecompressionBombError(
1543 "document package part has a malformed comment "
1544 "before its root element"
1545 )
1546 at += 1
1547 self._state = "comment"
1548 elif self._state == "comment":
1549 end = data.find(b"--", at)
1550 if end < 0:
1551 if data.endswith(b"-") and len(data) - 1 >= at:
1552 self._carry = b"-"
1553 return
1554 if end + 2 >= len(data):
1555 self._carry = data[end:]
1556 return
1557 if data[end + 2 : end + 3] != b">":
1558 raise DecompressionBombError(
1559 "document package part has a malformed comment "
1560 "before its root element"
1561 )
1562 at = end + 3
1563 self._state = "prolog"
1564 else:
1565 start = data.find(b"<", at)
1566 if start < 0:
1567 return
1568 if len(data) - start < 3:
1569 # "<" or "<!" (no declaration yet) reach the parser
1570 # and are decided with the next chunk.
1571 self._carry = data[start:]
1572 return
1573 after = data[start + 1 : start + 2]
1574 if after == b"?":
1575 self._state = "pi"
1576 at = start + 2
1577 elif after != b"!":
1578 self._state = "done"
1579 return
1580 elif data[start + 2 : start + 3] == b"-":
1581 self._state = "dash"
1582 at = start + 3
1583 else:
1584 raise DecompressionBombError(
1585 "document package part declares a DTD"
1586 )
1589class _CountedXmlStream:
1590 """A binary stream over *stream* (a member's stream or a
1591 ``BytesIO``) that an lxml ``iterparse`` reads in place of it: the
1592 part is held to ``_require_utf8_xml`` before its first byte is
1593 returned, and every chunk is counted by ``_XmlTagSigns`` and
1594 scanned by ``_XmlProlog`` before it is returned, so the parser
1595 never receives a start tag of more than
1596 ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` attributes or more of a document
1597 type declaration than the ``<!`` that opens it (when a chunk ends
1598 there; the next chunk is refused). A read returns at most
1599 ``_XML_SCAN_CHUNK`` bytes."""
1601 def __init__(self, stream) -> None:
1602 self._stream = stream
1603 self._signs: Optional[_XmlTagSigns] = None
1604 self._prolog = _XmlProlog()
1605 self._pending = b""
1607 def read(self, size: Optional[int] = -1) -> bytes:
1608 if self._signs is None:
1609 head = self._stream.read(_XML_SCAN_CHUNK)
1610 _require_utf8_xml(head)
1611 self._signs = _XmlTagSigns()
1612 self._signs.feed(head)
1613 self._prolog.feed(head)
1614 self._pending = head
1615 if size is None or size < 0 or size > _XML_SCAN_CHUNK: 1615 ↛ 1616line 1615 didn't jump to line 1616 because the condition on line 1615 was never true
1616 size = _XML_SCAN_CHUNK
1617 if self._pending:
1618 chunk = self._pending[:size]
1619 self._pending = self._pending[size:]
1620 return chunk
1621 chunk = cast(bytes, self._stream.read(size))
1622 self._signs.feed(chunk)
1623 self._prolog.feed(chunk)
1624 return chunk
1627def _read_counted_xml(archive: zipfile.ZipFile, info: zipfile.ZipInfo) -> bytes:
1628 """Member *info*, a whole part about to be parsed from a buffer,
1629 read through ``_CountedXmlStream``: held to ``_require_utf8_xml``,
1630 counted by ``_XmlTagSigns`` and scanned by ``_XmlProlog`` chunk by
1631 chunk as it is read, so a refused part is refused before the rest
1632 of it is read."""
1633 with archive.open(info) as member:
1634 stream = _CountedXmlStream(member)
1635 return b"".join(
1636 iter(functools.partial(stream.read, _XML_SCAN_CHUNK), b"")
1637 )
1640def _check_xlsx_sheet_fan_out(content: bytes) -> None:
1641 """Refuse a workbook listing the same sheet part more than once.
1643 openpyxl (via pandas) reads and converts a sheet part once per
1644 ``<sheet>`` that resolves to it, so N ``<sheet>`` entries sharing
1645 one part turn one bounded part into N times the parsing work and
1646 extracted text. Genuine workbooks never share a sheet part.
1648 The sheet list is computed by openpyxl itself — the same
1649 ``ExcelReader`` steps ``load_workbook`` runs (``read_manifest``,
1650 ``read_workbook``, ``WorkbookParser.find_sheets``) with the options
1651 pandas passes (``read_only``, ``data_only``, ``keep_links=False``,
1652 no VBA) — so the guard cannot disagree with the loader about which
1653 part each sheet names. Those steps parse only ``[Content_Types].xml``,
1654 the workbook part and the workbook's relationships (external links
1655 are not read with ``keep_links=False``); no worksheet, drawing or
1656 shared-string part is parsed. Each of those parts is first held to
1657 ``MAX_XLSX_SKELETON_PART_BYTES`` (16 MiB), scanned
1658 (``_check_xlsx_namespaces``, ``[Content_Types].xml`` with expat as
1659 well, for msoffcrypto) and checked before openpyxl parses it: a DTD
1660 is refused, and so, after one lxml parse
1661 of the part, is a comment or processing instruction below its root's
1662 children (``_refuse_xlsx_nested_node``; openpyxl would build one
1663 into a list entry), and the workbook part's lists and elements are
1664 held as ``_check_xlsx_workbook_tree`` holds them. The cost is two
1665 extra parses of at most that
1666 much XML per part, the same parse the loader performs right after.
1667 Once the sheet list passes (``_check_xlsx_sheet_list``), every other
1668 part openpyxl's load parses (``_xlsx_loaded_parts``) is scanned the
1669 same way, each with the parser the load uses for it, and the styles
1670 part is then streamed once (``_check_xlsx_styles``).
1672 ``rel.target`` is compared exactly: ``read_worksheets`` looks it up
1673 by exact name among the archive's members, so only identical
1674 strings load the same member. Every ``<sheet>``'s name, with or
1675 without a relationship, is held to ``MAX_XLSX_SHEET_NAME_CHARS``,
1676 and no two may be equal once lowercased.
1677 """
1678 try:
1679 import openpyxl # noqa: F401
1680 except ImportError: # no openpyxl, no openpyxl fan-out
1681 return
1683 try:
1684 # Imported inside the fail-closed block: if an openpyxl upgrade
1685 # moves these, every .xlsx is refused (and the control tests
1686 # fail) rather than the check silently switching off.
1687 from openpyxl.packaging.relationship import get_rels_path
1688 from openpyxl.reader.excel import ExcelReader, _find_workbook_part
1689 from openpyxl.xml.constants import ARC_CONTENT_TYPES
1691 reader = ExcelReader(
1692 io.BytesIO(content),
1693 read_only=True,
1694 keep_vba=False,
1695 data_only=True,
1696 keep_links=False,
1697 )
1698 with reader.archive:
1699 archive = reader.archive
1700 scanned: set[tuple[str, str]] = set()
1701 # The parts parsed here are package skeleton, a few KB in
1702 # real workbooks; a tighter ceiling than the general parsed
1703 # part keeps this extra full-tree parse cheap. Each is
1704 # scanned (_check_xlsx_namespaces) before it is parsed, and
1705 # [Content_Types].xml for msoffcrypto's minidom too.
1706 _require_skeleton_size(
1707 archive, ARC_CONTENT_TYPES, scanned, (_LXML, _EXPAT)
1708 )
1709 reader.read_manifest()
1710 workbook = _find_workbook_part(reader.package).PartName[1:]
1711 _require_skeleton_size(archive, workbook, scanned, workbook=True)
1712 _require_skeleton_size(archive, get_rels_path(workbook), scanned)
1713 reader.read_workbook()
1714 sheets = list(reader.parser.find_sheets())
1715 _check_xlsx_sheet_list(
1716 [rel.target for _sheet, rel in sheets],
1717 [sheet.name or "" for sheet in reader.parser.sheets],
1718 )
1719 _check_xlsx_namespaces(
1720 archive, _xlsx_loaded_parts(reader, sheets), scanned
1721 )
1722 _check_xlsx_styles(archive)
1723 except DecompressionBombError:
1724 raise
1725 except Exception as exc: # openpyxl's failure modes are open-ended
1726 raise DecompressionBombError("malformed spreadsheet package") from exc
1729def _check_xlsx_sheet_list(targets: list[str], names: list[str]) -> None:
1730 """Refuse the sheet list ``_check_xlsx_sheet_fan_out`` read (each
1731 sheet's resolved part and every sheet's name) for the reasons its
1732 docstring gives."""
1733 from openpyxl.packaging.relationship import get_rels_path
1735 if len(targets) > MAX_XLSX_SHEETS: 1735 ↛ 1736line 1735 didn't jump to line 1736 because the condition on line 1735 was never true
1736 raise DecompressionBombError(
1737 f"spreadsheet lists more than {MAX_XLSX_SHEETS} sheets"
1738 )
1739 if any(len(name) > MAX_XLSX_SHEET_NAME_CHARS for name in names):
1740 raise DecompressionBombError(
1741 "spreadsheet sheet name is longer than "
1742 f"{MAX_XLSX_SHEET_NAME_CHARS} characters"
1743 )
1744 folded = [name.lower() for name in names]
1745 if len(folded) != len(set(folded)):
1746 raise DecompressionBombError("spreadsheet lists two sheets of one name")
1747 if len(targets) != len(set(targets)):
1748 raise DecompressionBombError(
1749 "spreadsheet lists the same sheet part more than once"
1750 )
1751 # openpyxl also parses each sheet's own relationships part, found
1752 # via get_rels_path — which collapses targets that differ only in
1753 # repeated slashes onto one path — so those must be unique too.
1754 rels_paths = [get_rels_path(target) for target in targets]
1755 if len(rels_paths) != len(set(rels_paths)):
1756 raise DecompressionBombError(
1757 "spreadsheet sheets share one relationships part"
1758 )
1761def _xlsx_loaded_parts(reader, sheets) -> dict[str, frozenset[str]]:
1762 """The parts openpyxl's read-only load (``ExcelReader.read`` with
1763 the options pandas passes) parses after the workbook skeleton, each
1764 with its parser family, found as ``read`` finds them: the shared
1765 strings part the manifest names (``read_strings``, expat), the core
1766 and custom document properties and the styles part when present
1767 (``read_properties``, ``read_custom``, ``apply_stylesheet``, lxml),
1768 and for every sheet whose resolved part exists
1769 (``read_worksheets``) its relationships part when present
1770 (``get_dependents``, lxml) and the part itself: a chartsheet's
1771 (``"chartsheet"`` in the relationship type) with lxml
1772 (``read_chartsheet``), a worksheet's with expat, which pandas
1773 streams it with. ``read_theme`` reads the theme's bytes without
1774 parsing them, and drawings are never read
1775 (``openpyxl_hardening``), so neither is listed, nor is any part
1776 found only by its name or folder (images, unreferenced members)."""
1777 from openpyxl.packaging.relationship import get_rels_path
1778 from openpyxl.xml.constants import (
1779 ARC_CORE,
1780 ARC_CUSTOM,
1781 ARC_STYLE,
1782 SHARED_STRINGS,
1783 )
1785 valid = set(reader.valid_files)
1786 parts: dict[str, set[str]] = {}
1788 def add(name: str, family: str) -> None:
1789 if name in valid:
1790 parts.setdefault(name, set()).add(family)
1792 strings = reader.package.find(SHARED_STRINGS)
1793 if strings is not None:
1794 add(strings.PartName[1:], _EXPAT)
1795 for name in (ARC_CORE, ARC_CUSTOM, ARC_STYLE):
1796 add(name, _LXML)
1797 for _sheet, rel in sheets:
1798 if rel.target not in valid:
1799 continue
1800 add(get_rels_path(rel.target), _LXML)
1801 add(rel.target, _LXML if "chartsheet" in rel.Type else _EXPAT)
1802 return {name: frozenset(families) for name, families in parts.items()}
1805#: Ceiling for the package-skeleton parts the fan-out check parses in
1806#: full ([Content_Types].xml, the workbook part and its relationships).
1807#: Typical workbooks keep them in kilobytes, and thousands of sheets
1808#: stay well under 1 MiB, but the workbook part grows with defined
1809#: names: a workbook with more than roughly 170k defined names (about
1810#: 100 bytes each) exceeds 16 MiB and is refused. At the general
1811#: 128 MiB parsed-part ceiling, one extra full-tree parse would cost
1812#: gigabytes.
1813MAX_XLSX_SKELETON_PART_BYTES = 16 * 1024 * 1024
1815#: Ceiling on the sheets a workbook lists (``<sheet>`` entries with a
1816#: relationship id, as openpyxl's ``find_sheets`` yields them). openpyxl
1817#: checks each sheet's target against the archive's member list, a
1818#: Python list (``rel.target not in self.valid_files``), before it skips
1819#: a sheet whose part is missing, and pandas looks every sheet up by
1820#: name in the list of all sheet names, then in the workbook's sheets
1821#: (``raise_if_bad_sheet_by_name`` and ``Workbook.__getitem__``): sheets
1822#: times members, and sheets squared. 10,000 empty sheets, a 2.5 MB file
1823#: within the entry ceiling, took 40 s in ``pandas.read_excel`` before
1824#: the change, and 100,000 sheets without parts next to 9,000 other
1825#: members, a 2.2 MB file, 7.6 s in ``openpyxl.load_workbook`` alone
1826#: (a 16 MiB workbook part lists about 370,000). 2,000 empty sheets take
1827#: about 2 s in ``pandas.read_excel``, and 2,000 one-cell sheets about
1828#: 5 s in unstructured's ``partition_xlsx`` (500, 1.2 s). Workbooks
1829#: rarely have more than a few hundred sheets.
1830MAX_XLSX_SHEETS = 2_000
1832#: Ceiling on the characters of a sheet's name (``<sheet name>``).
1833#: openpyxl names every chartsheet it reads after comparing its name,
1834#: lowercased, with every sheet name before it, and pandas looks every
1835#: worksheet up by name in lists of all of them, so long names multiply
1836#: the per-sheet work ``MAX_XLSX_SHEETS`` bounds: 2,000 sheets of
1837#: 8,000-character names, a 554 KB file, took 6.4 s in
1838#: ``pandas.read_excel`` (of 31 characters, about 2 s). Excel limits a
1839#: sheet name to 31 characters and Google Sheets to 100. Two sheets
1840#: whose names are equal once lowercased (``str.lower``) are refused
1841#: too: Excel forbids them, and openpyxl renames such a chartsheet by
1842#: joining every sheet name into one string and searching it with a
1843#: regular expression, once per chartsheet.
1844MAX_XLSX_SHEET_NAME_CHARS = 255
1846#: Ceiling on the characters of a defined name's value in the workbook
1847#: part (measured at the longest of a ``definedName`` entry's attribute
1848#: values, its text and its children's: openpyxl reads the value from
1849#: an ``attr_text`` or ``attr-text`` attribute or the text, and other
1850#: fields from attributes or children of their names). Excel limits a
1851#: formula, and the reference a name stands for, to 8,192 characters.
1852MAX_XLSX_DEFINED_NAME_CHARS = 8_192
1854#: Ceiling on the work openpyxl's load spends on the print titles and
1855#: print areas of the workbook's defined names, each counted at its
1856#: value's length squared plus ``XLSX_PRINT_NAME_BASE_UNITS``. On every
1857#: load (pandas' read-only one too, ``WorkbookParser.assign_names``)
1858#: openpyxl parses each sheet-local name that starts like
1859#: ``_xlnm.Print_Titles`` or ``_xlnm.Print_Area`` (a prefix match, so
1860#: one sheet can hold any number of them) with
1861#: ``PrintTitles.from_string`` or ``PrintArea.from_string``, whose
1862#: regular expressions scan for a sheet title from every position of
1863#: the value and back off at its end: time quadratic in its length, up
1864#: to about 34 ns per character squared (a value of 40,000 ``#`` took
1865#: 14 s and 31 s, and a 1 MB one would take hours, holding the GIL),
1866#: plus about 50 us per name (65,536 values of 32 characters took
1867#: 5.5 s). The guard counts such names whatever their sheet. This
1868#: ceiling admits one value at ``MAX_XLSX_DEFINED_NAME_CHARS``, which
1869#: takes 2.3 to 2.9 s to load, 16,476 of 45 characters (2.2 s) or
1870#: 31,776 of 8 (0.9 s); Excel writes one print area and one set of
1871#: print titles per sheet, a sheet reference per range (under 100
1872#: characters each).
1873XLSX_PRINT_NAME_BASE_UNITS = 2_048
1874MAX_XLSX_PRINT_NAME_WORK = (
1875 MAX_XLSX_DEFINED_NAME_CHARS**2 + XLSX_PRINT_NAME_BASE_UNITS
1876)
1877_XLSX_PRINT_NAMES = ("_xlnm.Print_Titles", "_xlnm.Print_Area")
1879#: Ceiling on the characters of a namespace URI declared in any XML
1880#: member of an ``.xlsx`` package. lxml builds every element and
1881#: attribute name it hands back as ``{URI}local``, and openpyxl (and
1882#: the guard) read the name of every element of every part they parse,
1883#: so one long URI declared once and used through a short prefix is
1884#: copied again for every element: 2,000 empty elements under a
1885#: 1,000,000-character URI in a worksheet, a 5.9 KB file, took 3.8 s
1886#: in ``pandas.read_excel``. Office writers declare URIs well under 100
1887#: characters; this is the ``.docx`` ceiling
1888#: (``MAX_DOCX_NAMESPACE_URI_CHARS``). The scan that checks it in
1889#: every part openpyxl parses (``_check_xlsx_namespaces``), with a
1890#: Python call per element for the depth ceiling, costs the guard about
1891#: 6.5 s per 100 MB of worksheet XML (pandas takes about 20 s to read it)
1892#: and about 8 s per 100 MB of element-dense lxml parts.
1893MAX_XLSX_NAMESPACE_URI_CHARS = 1_024
1896#: Ceiling on the characters of a custom number format code
1897#: (``<numFmt formatCode>``) in ``xl/styles.xml``. Excel's format
1898#: dialog and the ``.xls`` Format record hold a code to 255
1899#: characters; this allows four times that.
1900MAX_XLSX_NUMBER_FORMAT_CHARS = 1_024
1902#: Ceiling on the entries of each of the cell-format lists of
1903#: ``xl/styles.xml`` (``<xf>`` elements of ``cellXfs`` and of
1904#: ``cellStyleXfs``, ``<cellStyle>`` elements of ``cellStyles``), each
1905#: counted over every such list in the part. Excel's specifications
1906#: limit a workbook to 65,490 unique cell formats and cell styles.
1907MAX_XLSX_CELL_FORMATS = 65_490
1909#: Ceiling on the work openpyxl's stylesheet load spends on number
1910#: formats, which it does on every load (``apply_stylesheet``, under
1911#: pandas' read-only load too). ``Stylesheet._normalise_numbers`` runs
1912#: ``is_date_format`` and ``is_timedelta_format`` (a split and three
1913#: regular-expression scans) over the format code of every ``<xf>`` of
1914#: ``cellXfs``, so N formats sharing one long code cost N times its
1915#: length: one 1,000,000-character code under 2,000 formats, a 3.3 KB
1916#: file, took 53 s to load. One scan skips ahead past a ``[`` only at
1917#: the next ``]``, so a code with B unclosed ``[`` costs up to B times
1918#: its length. Each format is counted at its code's length times
1919#: ``XLSX_NUMBER_FORMAT_SCAN_WEIGHT`` plus its ``[`` (a code shared by
1920#: several ``numFmt`` entries at the costliest of them; a built-in
1921#: format at 64 characters); a unit measured about 0.5 ns for codes of
1922#: zeros and of brackets alike: 3,600 formats of 1,024 ``[``, counted
1923#: at 0.99 of this ceiling, load in 2.1 s, and 65,490 formats of 1,024
1924#: zeros, at 0.84, in 2.9 s. 65,490 formats of an ordinary 30-character
1925#: date code with one bracket count about 100 million.
1926MAX_XLSX_NUMBER_FORMAT_WORK = 4_000_000_000
1927XLSX_NUMBER_FORMAT_SCAN_WEIGHT = 50
1928_XLSX_BUILTIN_FORMAT_CHARS = 64
1930#: Ceiling on the work openpyxl spends per named style, which it does
1931#: on every load (under pandas' read-only load too). For every named
1932#: style (``<cellStyle>``, duplicates dropped) ``Stylesheet``
1933#: builds a ``NamedStyle`` with fresh default font, fill, border,
1934#: alignment and protection objects, rebuilds the dict of every custom
1935#: number format (``custom_formats``) when its ``cellStyleXfs`` format
1936#: is custom, and ``apply_stylesheet`` binds it to the workbook
1937#: (``NamedStyle.bind``), which hashes its font, fill, border,
1938#: alignment, protection and number format into the workbook's indexed
1939#: lists and compares each with an equal entry already listed. A hash
1940#: walks the whole object, so a gradient fill costs every stop, and a
1941#: comparison reads every string, so a font costs its name's length.
1942#: Each named style is counted at ``XLSX_NAMED_STYLE_BASE_UNITS`` plus
1943#: the ``numFmt`` entries plus, for the largest font, the largest fill
1944#: and the largest border of the part (whichever one it binds),
1945#: ``XLSX_STYLE_NODE_UNITS`` per element and one unit per
1946#: ``XLSX_STYLE_CHARS_PER_UNIT`` characters of attribute values and
1947#: text, plus the longest format code at the same rate. A unit is
1948#: about 130 ns (one entry of the rebuilt format dict); an element
1949#: (one gradient stop is two) about 1.5 us, and the fixed work about
1950#: 75 us. Named styles are counted as the distinct ``xfId`` values of
1951#: ``cellStyle`` entries within ``-n`` to ``n - 1`` for the ``n``
1952#: ``cellStyleXfs`` entries: openpyxl drops a style whose ``xfId``
1953#: repeats, indexes a negative one from the end (so ``n`` formats
1954#: carry up to ``2n`` named styles) and fails to load an id out of that
1955#: range. 8,800 named styles over 4,400 formats and 4,400 custom
1956#: formats took 5.3 s to load, and 1,000 named styles of a 1,000-stop
1957#: gradient fill about 3 s; 20,000 named styles of a minimal font, fill
1958#: and border, counted at 0.66 of this ceiling, load in 1.8 s, 2,000 of
1959#: a 256-stop gradient fill, at 0.89, in 1.6 s, and a workbook bloated
1960#: to 15,000 named styles of Excel's default font, fills and border
1961#: with 300 custom formats, at 0.84, in 2 s (20,000 such, at 1.12,
1962#: take 2.7 s and are refused).
1963MAX_XLSX_NAMED_STYLE_WORK = 20_000_000
1964XLSX_NAMED_STYLE_BASE_UNITS = 600
1965XLSX_STYLE_NODE_UNITS = 16
1966XLSX_STYLE_CHARS_PER_UNIT = 256
1968#: Ceiling on the ``<stop>`` elements of one gradient fill in
1969#: ``xl/styles.xml``. openpyxl hashes every stop each time a named style
1970#: binds the fill. Excel's cell gradient dialog sets two or three
1971#: stops (its "Fill Effects" offer two colours); this allows 256.
1972MAX_XLSX_GRADIENT_STOPS = 256
1974_XLSX_STYLES_PART = "xl/styles.xml"
1975#: The ``xl/styles.xml`` lists whose entries openpyxl binds once per
1976#: named style (each read as a list of all the container's children,
1977#: whatever their tag; comments and processing instructions are refused
1978#: there, see ``_refuse_xlsx_nested_node``).
1979_XLSX_BOUND_LISTS = ("fonts", "fills", "borders")
1982def _refuse_xlsx_nested_node(depth: int, part: str) -> None:
1983 """Refuse a comment or processing instruction of the ``.xlsx``
1984 *part* (named for the message) that is *depth* elements deep,
1985 unless it lies outside the root (*depth* 0) or directly under it
1986 (*depth* 1).
1988 openpyxl parses ``xl/styles.xml`` and the workbook skeleton parts
1989 with lxml, which keeps comments and processing instructions as
1990 child nodes, and ``NestedSequence`` builds one entry per child
1991 *node* of its container, whatever it is
1992 (``[expected_type.from_tree(el) for el in node]``): a comment among
1993 the ``fonts``, ``borders`` or ``dxfs`` of ``xl/styles.xml`` becomes a
1994 default ``Font``, ``Border`` or ``DifferentialStyle``, one among
1995 ``mruColors`` a default ``Color``, and one among the workbook's
1996 ``bookViews`` a default ``BookView`` (among ``fills``,
1997 ``indexedColors`` and ``sheets`` openpyxl fails to load instead).
1998 The guard counts elements, so 2,000,000 empty comments after one
1999 font, a 25 KB file whose styles part is 14 MB, were accepted and
2000 took openpyxl 37 s and 960 MB to load as 2,000,001 fonts, and
2001 2,390,000 among the ``bookViews`` of a 16 MiB workbook part took the
2002 guard's own openpyxl parse of it 19.6 s and 1.1 GB. Elsewhere below
2003 the root openpyxl skips them (``Serialisable.from_tree``, which
2004 reads ``Sequence`` lists such as ``cellXfs``, ``numFmts``,
2005 ``cellStyles`` or a gradient's stops, and single entries) or fails
2006 to load (in a ``fill`` before its pattern, or in a ``patternFill``).
2007 Directly under the root ``Serialisable.from_tree`` skips them too,
2008 at the cost of the tree node lxml builds for each, and the
2009 relationships list, read as every child node of its root, fails to
2010 load, which leaves openpyxl without the workbook's sheets (refused
2011 as malformed). Excel writes no comments or processing instructions
2012 in these parts, so any below the root's children is refused rather
2013 than counted, and every child node of a list the guard counts is
2014 then an element it counts."""
2015 if depth >= 2:
2016 raise DecompressionBombError(
2017 f"spreadsheet {part} part has a comment or processing "
2018 "instruction below its top level"
2019 )
2022#: Ceiling on the entries of each of the font, fill and border lists of
2023#: ``xl/styles.xml``, each counted over every such list in the part.
2024#: openpyxl builds every entry into a style object and an indexed list
2025#: of each list on every load, and the guard rebuilds each differently
2026#: spelled entry once (``MAX_XLSX_STYLE_HASH_CHAIN``). Excel's
2027#: specifications allow 512 fonts per workbook and 256 fill and line
2028#: styles; this allows 8,192 of each. A 16 MB styles part with all
2029#: three lists at this ceiling and 65,490 cell formats, each with its
2030#: own alignment and protection, loads in 5.3 s (the guard takes 10 s).
2031MAX_XLSX_STYLE_LIST_ENTRIES = 8_192
2033#: Ceilings on the other lists of ``xl/styles.xml`` and of the workbook
2034#: part that openpyxl builds an object from per entry, each counted over
2035#: every such list in the part (``_XLSX_STYLE_LISTS`` and
2036#: ``_XLSX_WORKBOOK_LISTS``). Excel documents 200 to 250 custom number
2037#: formats per workbook; the differential formats of conditional
2038#: formatting and tables (``dxfs``) are held to its 65,490 cell formats;
2039#: Excel writes a 64-colour ``indexedColors`` palette and up to ten
2040#: ``mruColors``; custom table styles, workbook views, external
2041#: references and pivot caches have no documented limit and number a
2042#: handful in real files. A table style has one element per table part
2043#: (28 kinds; counted per table style). Extensions (``<ext>`` of every
2044#: ``extLst``, counted over the part) number one per cell format at most
2045#: in Excel's output (``xfComplement``), plus a few.
2046MAX_XLSX_NUMBER_FORMATS = 8_192
2047MAX_XLSX_DIFFERENTIAL_FORMATS = 65_490
2048MAX_XLSX_COLOR_LIST_ENTRIES = 1_024
2049MAX_XLSX_TABLE_STYLES = 1_024
2050MAX_XLSX_TABLE_STYLE_ELEMENTS = 32
2051MAX_XLSX_EXTENSIONS = 131_072
2052MAX_XLSX_WORKBOOK_LIST_ENTRIES = 8_192
2054#: Ceiling on the elements below ``dxfs`` in ``xl/styles.xml`` (its
2055#: entries and everything in them). A differential format holds up to a
2056#: font, number format, fill, alignment, border and protection, each a
2057#: handful of elements, and openpyxl builds them all: 10,000 with every
2058#: one of those filled in (43 elements each, a 9 MB part) take 3.5 s to
2059#: load. Excel writes a few elements per differential format.
2060MAX_XLSX_DIFFERENTIAL_FORMAT_ELEMENTS = 500_000
2062#: Ceiling on the gradient ``<stop>`` elements of ``xl/styles.xml``
2063#: together (fills and differential formats). openpyxl builds every
2064#: stop and its colour, and the guard rebuilds every distinct fill:
2065#: 1,000 fills of 256 stops, a 14 MB part, took 4.5 s to load and the
2066#: guard 8 s, and the 8,192 fills allowed would take about 37 s and
2067#: 66 s. Excel's gradients have two or three stops.
2068MAX_XLSX_GRADIENT_STOPS_TOTAL = 65_536
2070#: Ceiling on the elements of one entry of the font, fill, border and
2071#: cell-format lists of ``xl/styles.xml`` (the entry and everything in
2072#: it, the content of an extension included). The guard holds each such
2073#: entry whole until it ends, to rebuild it with openpyxl's classes,
2074#: and then drops it; lxml frees a detached subtree in linear time, but
2075#: when a Python proxy of a node inside it is still alive (as
2076#: ``iterparse`` leaves some) and the part declares a namespace, as
2077#: every Office part does, lxml instead moves the subtree to a document
2078#: of its own in time quadratic in its nodes: one fill holding 80,000
2079#: elements took the guard 1.6 s to drop, 160,000 took 6.5 s. Excel
2080#: writes a dozen or so; a gradient fill at ``MAX_XLSX_GRADIENT_STOPS``
2081#: stops has 514.
2082MAX_XLSX_STYLE_ENTRY_ELEMENTS = 1_024
2084#: Ceiling on the child elements of the root of ``xl/styles.xml`` and of
2085#: the workbook part. openpyxl builds every child it knows into an
2086#: object (keeping only the last of a repeated one), so 1,800,000 empty
2087#: ``<calcPr/>`` in a 16 MiB workbook part took its parse 17 s (the
2088#: guard's own parse as long again). Excel writes about a dozen.
2089MAX_XLSX_TOP_LEVEL_ELEMENTS = 256
2091#: Range of the integers openpyxl reads from ``xl/styles.xml`` (ids,
2092#: colour indices, charsets and the like; the schema types them as
2093#: 32-bit, ``xsd:int`` or ``xsd:unsignedInt``), and the magnitude of its
2094#: floats (sizes, tints, gradient geometry; Excel's largest font is 409
2095#: points). Python hashes a number modulo ``2**61 - 1``, so integers
2096#: ``1 + k * (2**61 - 1)`` are distinct values of one hash: 4,000 fonts
2097#: differing only in such a ``charset`` took 6.6 s to load (a 51 KB
2098#: part) and 8,000 of them under 8,000 named styles 81 s. Within this
2099#: range distinct integers hash apart, which also keeps the dicts
2100#: openpyxl and the guard key by format and style ids linear.
2101XLSX_STYLE_INT_MIN = -(2**31)
2102XLSX_STYLE_INT_MAX = 2**32 - 1
2103MAX_XLSX_STYLE_FLOAT = float(2**32)
2105#: Ceiling on the distinct values of one hash in any of the indexed
2106#: lists openpyxl's stylesheet load builds (fonts, fills, borders,
2107#: alignments, protections, cell formats, number format codes) and in
2108#: its set of named-style names. Each list is a dict keyed by the
2109#: style object, whose hash openpyxl takes over the object's field
2110#: values, so distinct objects of one hash cost a comparison with each
2111#: other on insertion (quadratic in their number) and on every later
2112#: lookup, which each named style repeats for its font, fill, border,
2113#: alignment and protection. Bounding the numbers alone does not
2114#: prevent it: a float field reaches almost any hash value (``x`` and
2115#: ``x * 2**-61`` share one), and the hash of a tuple of free 32-bit
2116#: fields can be steered by a meet-in-the-middle search. Real
2117#: stylesheets have none; this allows the pair Python's hash folds
2118#: (``-1`` and ``-2``). 28,000 named styles each binding the second of
2119#: a pair in every list loaded in 8.2 s, against 7 s without pairs.
2120MAX_XLSX_STYLE_HASH_CHAIN = 2
2122#: The lists of ``xl/styles.xml`` under its root (matched by local
2123#: name) and under its ``colors``, each read as a list of entries: by
2124#: ``NestedSequence`` (every child node) or ``Sequence`` (every child of
2125#: the entry's name; ``cellXfs`` also reads ``alignment`` and
2126#: ``protection`` children), all counted here.
2127_XLSX_STYLE_LISTS = (
2128 "numFmts",
2129 "fonts",
2130 "fills",
2131 "borders",
2132 "cellStyleXfs",
2133 "cellXfs",
2134 "cellStyles",
2135 "dxfs",
2136 "tableStyles",
2137)
2138_XLSX_COLOR_LISTS = ("indexedColors", "mruColors")
2140#: The lists of the workbook part under its root, each counted over
2141#: every such list in the part: ``NestedSequence`` lists and the
2142#: ``Sequence`` lists of the smart tag, function group and web publish
2143#: objects. ``definedNames`` is held only by the part's 16 MiB ceiling
2144#: (``MAX_XLSX_SKELETON_PART_BYTES``).
2145_XLSX_WORKBOOK_LISTS = (
2146 "bookViews",
2147 "sheets",
2148 "customWorkbookViews",
2149 "externalReferences",
2150 "pivotCaches",
2151 "smartTagTypes",
2152 "functionGroups",
2153 "webPublishObjects",
2154)
2157def _xlsx_list_cap(name: str) -> int:
2158 """The ceiling on the entries of the ``.xlsx`` list *name*."""
2159 return {
2160 "numFmts": MAX_XLSX_NUMBER_FORMATS,
2161 "fonts": MAX_XLSX_STYLE_LIST_ENTRIES,
2162 "fills": MAX_XLSX_STYLE_LIST_ENTRIES,
2163 "borders": MAX_XLSX_STYLE_LIST_ENTRIES,
2164 "cellStyleXfs": MAX_XLSX_CELL_FORMATS,
2165 "cellXfs": MAX_XLSX_CELL_FORMATS,
2166 "cellStyles": MAX_XLSX_CELL_FORMATS,
2167 "dxfs": MAX_XLSX_DIFFERENTIAL_FORMATS,
2168 "tableStyles": MAX_XLSX_TABLE_STYLES,
2169 "indexedColors": MAX_XLSX_COLOR_LIST_ENTRIES,
2170 "mruColors": MAX_XLSX_COLOR_LIST_ENTRIES,
2171 "tableStyle": MAX_XLSX_TABLE_STYLE_ELEMENTS,
2172 "extLst": MAX_XLSX_EXTENSIONS,
2173 "sheets": MAX_XLSX_SHEETS,
2174 }.get(name, MAX_XLSX_WORKBOOK_LIST_ENTRIES)
2177def _refuse_xlsx_list_entry(part: str, name: str, count: int) -> None:
2178 """Refuse the ``.xlsx`` *part* once list *name* has *count* entries
2179 over its ceiling (``_xlsx_list_cap``)."""
2180 cap = _xlsx_list_cap(name)
2181 if count > cap:
2182 if name == "sheets":
2183 raise DecompressionBombError(
2184 f"spreadsheet lists more than {cap} sheets"
2185 )
2186 raise DecompressionBombError(
2187 f"spreadsheet {part} list more than {cap} {name} entries"
2188 )
2191def _refuse_xlsx_repeated_child(part: str, parent: str, seen: set, tag: str):
2192 """Refuse the ``.xlsx`` *part* when an element that is not a list
2193 (*parent*, whose child local names so far are *seen*) has a second
2194 child named *tag*. openpyxl builds every child it knows into an
2195 object and keeps only the last of a repeated one, so repeats cost a
2196 build each and are bounded by nothing else; Excel writes none."""
2197 if tag in seen:
2198 raise DecompressionBombError(
2199 f"spreadsheet {part} element {parent} repeats its {tag} child"
2200 )
2201 seen.add(tag)
2204#: The ``xl/styles.xml`` lists whose entries the guard rebuilds as
2205#: openpyxl's style objects (held in full until the entry ends).
2206_XLSX_MODELLED_LISTS = ("fonts", "fills", "borders", "cellXfs", "cellStyleXfs")
2209class _XlsxIndexedList:
2210 """The distinct values openpyxl's ``IndexedList`` of *what* holds,
2211 grouped by hash: ``add`` returns a value's index as
2212 ``IndexedList.add`` does and refuses a value that would be the
2213 ``MAX_XLSX_STYLE_HASH_CHAIN + 1``-th distinct one of its hash. The
2214 groups are keyed by hash values (ints), which reduce modulo
2215 ``2**61 - 1`` again, so at most eight of them share a slot here."""
2217 def __init__(self, what: str, initial: tuple = ()) -> None:
2218 self._what = what
2219 self._groups: dict[int, list[tuple[object, int]]] = {}
2220 self._size = 0
2221 for value in initial:
2222 self.add(value)
2224 def add(self, value) -> int:
2225 group = self._groups.setdefault(hash(value), [])
2226 for known, index in group:
2227 if known is value or known == value:
2228 return index
2229 if len(group) >= MAX_XLSX_STYLE_HASH_CHAIN:
2230 raise DecompressionBombError(
2231 f"spreadsheet styles hold more than "
2232 f"{MAX_XLSX_STYLE_HASH_CHAIN} distinct {self._what} of "
2233 "one hash"
2234 )
2235 group.append((value, self._size))
2236 self._size += 1
2237 return self._size - 1
2240@functools.lru_cache(maxsize=1)
2241def _xlsx_numeric_fields() -> tuple[
2242 frozenset[str], frozenset[str], frozenset[str]
2243]:
2244 """Names of the integer fields and of the float fields of the
2245 openpyxl style classes ``Stylesheet`` reads (walked from it through
2246 every complex field and its subclasses), and of those read from a
2247 ``val`` attribute of a child element of that name (``<sz val>``).
2248 Derived from openpyxl so an upgrade that adds a field is covered.
2249 No name is an integer in one class and a float in another."""
2250 from openpyxl.descriptors import Descriptor
2251 from openpyxl.descriptors.serialisable import Serialisable
2252 from openpyxl.styles.stylesheet import Stylesheet
2254 ints: set[str] = set()
2255 floats: set[str] = set()
2256 nested: set[str] = set()
2257 seen: set[type] = set()
2258 pending: list[type] = [Stylesheet]
2259 while pending:
2260 cls = pending.pop()
2261 if cls in seen:
2262 continue
2263 seen.add(cls)
2264 pending.extend(cls.__subclasses__())
2265 for name in dir(cls):
2266 desc = getattr(cls, name, None)
2267 if not isinstance(desc, Descriptor):
2268 continue
2269 kind = getattr(desc, "expected_type", None)
2270 if isinstance(kind, type) and issubclass(kind, Serialisable):
2271 pending.append(kind)
2272 elif kind is int or kind is float:
2273 (ints if kind is int else floats).add(name)
2274 if getattr(desc, "nested", False):
2275 nested.add(name)
2276 return frozenset(ints), frozenset(floats), frozenset(nested)
2279def _check_xlsx_style_number(name: str, value: Optional[str]) -> None:
2280 """Refuse *value* of the numeric style field *name* if openpyxl would
2281 read it as an integer outside ``XLSX_STYLE_INT_MIN`` to
2282 ``XLSX_STYLE_INT_MAX`` or a float of magnitude over
2283 ``MAX_XLSX_STYLE_FLOAT`` (NaN included). A value that is not a
2284 number is left to openpyxl, which fails to load it."""
2285 ints, floats, _nested = _xlsx_numeric_fields()
2286 if value is None or (name not in ints and name not in floats):
2287 return
2288 try:
2289 number: float = int(value) if name in ints else float(value)
2290 except ValueError:
2291 return
2292 if name in ints:
2293 fits = XLSX_STYLE_INT_MIN <= number <= XLSX_STYLE_INT_MAX
2294 else:
2295 fits = abs(number) <= MAX_XLSX_STYLE_FLOAT
2296 if not fits:
2297 raise DecompressionBombError(
2298 f"spreadsheet style {name} value is out of range"
2299 )
2302def _xlsx_text_fields(cls) -> frozenset[str]:
2303 """Names of the fields of openpyxl style class *cls* that
2304 ``Serialisable.from_tree`` also reads from a child element of that
2305 local name (its text), overriding the attribute: every descriptor
2306 that is neither a converter itself nor typed by a class with its
2307 own ``from_tree``. Derived from openpyxl so an upgrade that adds a
2308 field is covered; ``CellStyle`` gives ``numFmtId``, ``fontId``,
2309 ``fillId``, ``borderId``, ``xfId``, ``quotePrefix``,
2310 ``pivotButton`` and the ``apply*`` flags (its ``alignment``,
2311 ``protection`` and ``extLst`` children, which Excel writes, are
2312 complex types), ``_NamedCellStyle`` ``name``, ``xfId``,
2313 ``builtinId``, ``iLevel``, ``hidden`` and ``customBuiltin`` (its
2314 ``extLst`` is complex)."""
2315 from openpyxl.descriptors import Descriptor
2317 fields = set()
2318 for name in dir(cls):
2319 desc = getattr(cls, name, None)
2320 if (
2321 isinstance(desc, Descriptor)
2322 and not hasattr(desc, "from_tree")
2323 and hasattr(desc, "expected_type")
2324 and not hasattr(desc.expected_type, "from_tree")
2325 ):
2326 fields.add(name)
2327 return frozenset(fields)
2330def _check_xlsx_styles(archive: zipfile.ZipFile) -> None:
2331 """Refuse a workbook whose ``xl/styles.xml`` (the fixed name
2332 openpyxl reads it by) declares a DTD, has a number format code over
2333 ``MAX_XLSX_NUMBER_FORMAT_CHARS``, more than ``MAX_XLSX_CELL_FORMATS``
2334 entries in a cell-format list, a gradient fill of more than
2335 ``MAX_XLSX_GRADIENT_STOPS`` stops, a field the guard reads given as
2336 a child element, more than ``MAX_XLSX_STYLE_LIST_ENTRIES`` fonts,
2337 fills or borders, a numeric field out of the range
2338 ``_check_xlsx_style_number`` allows, more than
2339 ``MAX_XLSX_STYLE_HASH_CHAIN`` distinct style objects of one hash in
2340 any indexed list openpyxl builds from it, a comment or processing
2341 instruction below the root's children (``_refuse_xlsx_nested_node``)
2342 or more than ``MAX_XML_OUTSIDE_ROOT_NODES`` outside the root, more
2343 than ``MAX_XLSX_TOP_LEVEL_ELEMENTS`` elements under the root, more
2344 entries in its lists (``_XLSX_STYLE_LISTS``, ``_XLSX_COLOR_LISTS``,
2345 a table style's elements and every ``extLst``'s extensions, every
2346 child element counting) than ``_xlsx_list_cap`` allows, more than
2347 ``MAX_XLSX_DIFFERENTIAL_FORMAT_ELEMENTS`` elements under ``dxfs``,
2348 more than ``MAX_XLSX_GRADIENT_STOPS_TOTAL`` gradient stops in all,
2349 an element that is not a list repeating a child's local name
2350 (``_refuse_xlsx_repeated_child``; the content of an extension, which
2351 openpyxl does not read, excepted), a font, fill, border or
2352 cell-format entry of more than ``MAX_XLSX_STYLE_ENTRY_ELEMENTS``
2353 elements, or would cost openpyxl's stylesheet load more than
2354 ``MAX_XLSX_NUMBER_FORMAT_WORK`` or ``MAX_XLSX_NAMED_STYLE_WORK``.
2355 Elements are classed as openpyxl reads them: an ``extLst`` is an
2356 extension list only as the child of an element that is not a list
2357 (as a list's entry it is that entry, and as a fill's first child a
2358 gradient whose stops count), and only the ``ext`` children of an
2359 extension list are extensions.
2361 The part is read as openpyxl reads it: children are matched by
2362 local name in any namespace (``Serialisable.from_tree``) under any
2363 root, and numeric ids are read with ``int``. openpyxl keeps the last
2364 of repeated lists; every one is counted here. ``from_tree`` also
2365 reads a field from a child element of the field's name, whose text
2366 replaces the attribute (``<numFmt numFmtId="164"><formatCode>``),
2367 so any element child of a ``numFmt`` is refused, as is a child of a
2368 cell-format ``xf`` or a ``cellStyle`` named after one of its
2369 attribute fields (``_xlsx_text_fields``); Excel writes neither.
2370 The part is streamed once, each element dropped once read (an entry
2371 of the font, fill, border and cell-format lists once that entry has
2372 been rebuilt with openpyxl's own ``from_tree``), so the guard holds
2373 the path to the current element, the ``cellStyle`` ids, the cell
2374 formats' format ids and the distinct style objects (which openpyxl
2375 holds too). A non-integer id, which openpyxl cannot load, is refused
2376 as malformed."""
2377 from lxml import etree
2378 from openpyxl.styles.alignment import Alignment
2379 from openpyxl.styles.borders import Border
2380 from openpyxl.styles.cell_style import CellStyle
2381 from openpyxl.styles.fills import Fill
2382 from openpyxl.styles.fonts import Font
2383 from openpyxl.styles.named_styles import _NamedCellStyle
2384 from openpyxl.styles.protection import Protection
2386 try:
2387 info = archive.getinfo(_XLSX_STYLES_PART)
2388 except KeyError:
2389 return
2390 ints, floats, nested = _xlsx_numeric_fields()
2391 numeric = ints | floats
2392 entry_classes = {"fonts": Font, "fills": Fill, "borders": Border}
2393 # openpyxl's indexed lists, seeded as it seeds them.
2394 lists = {
2395 "fonts": _XlsxIndexedList("fonts"),
2396 "fills": _XlsxIndexedList("fills"),
2397 "borders": _XlsxIndexedList("borders"),
2398 }
2399 alignments = _XlsxIndexedList("alignments", (Alignment(),))
2400 protections = _XlsxIndexedList("protections", (Protection(),))
2401 cell_styles = _XlsxIndexedList("cell formats")
2402 strings = _XlsxIndexedList("format codes or style names")
2403 modelled: dict[str, set[bytes]] = {
2404 name: set() for name in _XLSX_MODELLED_LISTS
2405 }
2406 text_fields = {
2407 ("cellXfs", "xf"): _xlsx_text_fields(CellStyle),
2408 ("cellStyleXfs", "xf"): _xlsx_text_fields(CellStyle),
2409 ("cellStyles", "cellStyle"): _xlsx_text_fields(_NamedCellStyle),
2410 }
2411 format_units: dict[int, int] = {}
2412 cell_format_ids: list[int] = []
2413 style_ids: set[int] = set()
2414 counts = {"cellXfs": 0, "cellStyleXfs": 0, "cellStyles": 0}
2415 entry_tags = {
2416 "cellXfs": "xf",
2417 "cellStyleXfs": "xf",
2418 "cellStyles": "cellStyle",
2419 }
2420 bound_units = dict.fromkeys(_XLSX_BOUND_LISTS, 0)
2421 number_formats = longest_code = depth = 0
2422 entry_nodes = entry_chars = stops = outside = top = held = 0
2423 all_stops = dxf_elements = 0
2424 extensions = [0]
2425 path: list[str] = []
2426 # Per open element: ("root"), ("set", child names so far) for an
2427 # element that is not a list, ("list", name, counter) for a list,
2428 # ("gradient", child names but stops), or ("skip") inside an
2429 # extension, whose content openpyxl does not read.
2430 frames: list[tuple] = []
2431 totals = {name: [0] for name in _XLSX_STYLE_LISTS + _XLSX_COLOR_LISTS}
2432 with archive.open(info) as stream:
2433 for event, element in etree.iterparse(
2434 _CountedXmlStream(stream),
2435 events=("start", "end", "comment", "pi"),
2436 resolve_entities=False,
2437 no_network=True,
2438 ):
2439 if event in ("comment", "pi"):
2440 _refuse_xlsx_nested_node(depth, "styles")
2441 if depth == 0:
2442 outside = _note_outside_root_node(outside)
2443 # Outside the root or directly under it, where openpyxl
2444 # skips it: drop the siblings before it (it gets no end
2445 # event; the next sibling's end drops it in turn).
2446 parent = element.getparent()
2447 if parent is not None:
2448 while element.getprevious() is not None:
2449 del parent[0]
2450 continue
2451 if event == "end":
2452 tag = path[-1]
2453 # from_tree reads such a child's text (before its own
2454 # first child, if it has any) in place of the attribute.
2455 if (
2456 tag in numeric
2457 and tag not in nested
2458 and (element.text or "").strip()
2459 ):
2460 _check_xlsx_style_number(tag, element.text)
2461 if depth == 3 and path[1] in _XLSX_MODELLED_LISTS:
2462 # An entry spelled like one already rebuilt adds
2463 # nothing new (bloated workbooks repeat entries).
2464 spelling = hashlib.blake2b(
2465 etree.tostring(element), digest_size=16
2466 ).digest()
2467 if spelling not in modelled[path[1]]:
2468 modelled[path[1]].add(spelling)
2469 _model_xlsx_style_entry(
2470 element,
2471 path[1],
2472 entry_classes,
2473 lists,
2474 alignments,
2475 protections,
2476 cell_styles,
2477 )
2478 if depth >= 3 and path[1] in bound_units:
2479 entry_nodes += 1
2480 entry_chars += len(element.text or "") + sum(
2481 len(value) for value in element.attrib.values()
2482 )
2483 if depth == 3:
2484 units = (
2485 entry_nodes * XLSX_STYLE_NODE_UNITS
2486 + entry_chars // XLSX_STYLE_CHARS_PER_UNIT
2487 )
2488 if units > bound_units[path[1]]:
2489 bound_units[path[1]] = units
2490 entry_nodes = entry_chars = 0
2491 depth -= 1
2492 path.pop()
2493 frames.pop()
2494 if depth >= 3 and path[1] in _XLSX_MODELLED_LISTS:
2495 continue # rebuilt with its entry, then dropped
2496 element.clear()
2497 parent = element.getparent()
2498 if parent is not None:
2499 while element.getprevious() is not None:
2500 del parent[0]
2501 continue
2502 depth += 1
2503 tag = etree.QName(element).localname
2504 path.append(tag)
2505 if depth == 1:
2506 _refuse_dtd(element)
2507 frames.append(("root",))
2508 continue
2509 frame = frames[-1]
2510 kind = frame[0]
2511 if depth >= 3 and path[1] in _XLSX_MODELLED_LISTS:
2512 # Held whole until the entry ends, extension content
2513 # included (see MAX_XLSX_STYLE_ENTRY_ELEMENTS).
2514 held = 1 if depth == 3 else held + 1
2515 if held > MAX_XLSX_STYLE_ENTRY_ELEMENTS:
2516 raise DecompressionBombError(
2517 f"spreadsheet style {path[2]} entry has more than "
2518 f"{MAX_XLSX_STYLE_ENTRY_ELEMENTS} elements"
2519 )
2520 if kind == "skip":
2521 frames.append(frame)
2522 else:
2523 if kind == "root":
2524 top += 1
2525 if top > MAX_XLSX_TOP_LEVEL_ELEMENTS:
2526 raise DecompressionBombError(
2527 "spreadsheet styles part has more than "
2528 f"{MAX_XLSX_TOP_LEVEL_ELEMENTS} top-level elements"
2529 )
2530 elif kind == "list":
2531 frame[2][0] += 1
2532 _refuse_xlsx_list_entry("styles", frame[1], frame[2][0])
2533 elif kind == "gradient" and tag == "stop":
2534 stops += 1
2535 all_stops += 1
2536 if stops > MAX_XLSX_GRADIENT_STOPS:
2537 raise DecompressionBombError(
2538 "spreadsheet gradient fill has more than "
2539 f"{MAX_XLSX_GRADIENT_STOPS} stops"
2540 )
2541 if all_stops > MAX_XLSX_GRADIENT_STOPS_TOTAL:
2542 raise DecompressionBombError(
2543 "spreadsheet gradient fills have more than "
2544 f"{MAX_XLSX_GRADIENT_STOPS_TOTAL} stops together"
2545 )
2546 else:
2547 _refuse_xlsx_repeated_child(
2548 "styles", path[-2], frame[1], tag
2549 )
2550 if path[1] == "dxfs" and depth >= 3:
2551 dxf_elements += 1
2552 if dxf_elements > MAX_XLSX_DIFFERENTIAL_FORMAT_ELEMENTS:
2553 raise DecompressionBombError(
2554 "spreadsheet differential formats have more "
2555 f"than {MAX_XLSX_DIFFERENTIAL_FORMAT_ELEMENTS} "
2556 "elements"
2557 )
2558 # The kind of this element, as openpyxl reads it. It
2559 # reads a fill's first child, whatever its tag, as a
2560 # gradient unless that tag (namespace included) contains
2561 # "patternFill"; every child of a list as an entry
2562 # (``NestedSequence``, whatever its tag) or not at all
2563 # (``Sequence``, another tag), never as an extension
2564 # list; and an extension's content not at all.
2565 if tag == "ext" and kind == "list" and frame[1] == "extLst":
2566 frames.append(("skip",))
2567 elif (
2568 kind == "set"
2569 and len(frame[1]) == 1
2570 and (
2571 (depth == 4 and path[1] == "fills")
2572 or path[-2] == "fill"
2573 )
2574 and "patternFill" not in element.tag
2575 ):
2576 stops = 0
2577 frames.append(("gradient", set()))
2578 elif tag == "extLst" and kind != "list":
2579 frames.append(("list", tag, extensions))
2580 elif depth == 2 and tag in totals:
2581 frames.append(("list", tag, totals[tag]))
2582 elif depth == 3 and path[1] == "colors" and tag in totals:
2583 frames.append(("list", tag, totals[tag]))
2584 elif depth == 3 and path[1] == "tableStyles":
2585 frames.append(("list", "tableStyle", [0]))
2586 else:
2587 frames.append(("set", set()))
2588 for key, value in element.attrib.items():
2589 _check_xlsx_style_number(key, value)
2590 if tag in nested:
2591 _check_xlsx_style_number(tag, element.get("val"))
2592 if depth >= 3 and path[-2] == "numFmt":
2593 raise DecompressionBombError(
2594 "spreadsheet number format has a child element"
2595 )
2596 if depth == 4 and tag in text_fields.get((path[1], path[2]), ()):
2597 raise DecompressionBombError(
2598 f"spreadsheet style {path[2]} gives {tag} as a child "
2599 "element"
2600 )
2601 if depth != 3:
2602 continue
2603 container = path[1]
2604 if container == "numFmts" and tag == "numFmt":
2605 number_formats += 1
2606 code = element.get("formatCode") or ""
2607 strings.add(code)
2608 if len(code) > MAX_XLSX_NUMBER_FORMAT_CHARS:
2609 raise DecompressionBombError(
2610 "spreadsheet number format code is longer than "
2611 f"{MAX_XLSX_NUMBER_FORMAT_CHARS} characters"
2612 )
2613 longest_code = max(longest_code, len(code))
2614 format_id = int(element.get("numFmtId"))
2615 units = _xlsx_format_units(code)
2616 if units > format_units.get(format_id, 0):
2617 format_units[format_id] = units
2618 elif entry_tags.get(container) == tag:
2619 counts[container] += 1
2620 if container == "cellXfs":
2621 cell_format_ids.append(int(element.get("numFmtId", 0)))
2622 elif container == "cellStyles":
2623 style_ids.add(int(element.get("xfId")))
2624 strings.add(element.get("name"))
2625 builtin = _XLSX_BUILTIN_FORMAT_CHARS * XLSX_NUMBER_FORMAT_SCAN_WEIGHT
2626 work = sum(
2627 format_units.get(format_id, builtin) for format_id in cell_format_ids
2628 )
2629 if work > MAX_XLSX_NUMBER_FORMAT_WORK:
2630 raise DecompressionBombError(
2631 f"spreadsheet number formats cost {work} units in openpyxl's "
2632 f"stylesheet load (ceiling {MAX_XLSX_NUMBER_FORMAT_WORK})"
2633 )
2634 style_formats = counts["cellStyleXfs"]
2635 named_styles = sum(
2636 -style_formats <= style_id < style_formats for style_id in style_ids
2637 )
2638 per_style = (
2639 XLSX_NAMED_STYLE_BASE_UNITS
2640 + number_formats
2641 + sum(bound_units.values())
2642 + longest_code // XLSX_STYLE_CHARS_PER_UNIT
2643 )
2644 named = named_styles * per_style
2645 if named > MAX_XLSX_NAMED_STYLE_WORK:
2646 raise DecompressionBombError(
2647 f"spreadsheet named styles cost {named} units in openpyxl's "
2648 f"stylesheet load (ceiling {MAX_XLSX_NAMED_STYLE_WORK})"
2649 )
2652def _model_xlsx_style_entry(
2653 element,
2654 container: str,
2655 entry_classes: dict,
2656 lists: dict[str, _XlsxIndexedList],
2657 alignments: _XlsxIndexedList,
2658 protections: _XlsxIndexedList,
2659 cell_styles: _XlsxIndexedList,
2660) -> None:
2661 """Rebuild one entry of a styles list as openpyxl's load does and add
2662 it to the indexed lists it lands in: a font, fill or border entry
2663 (any tag; ``NestedSequence.from_tree``) to its list
2664 (``apply_stylesheet``); a cell format ``xf``'s alignment and
2665 protection to theirs and its style array to the cell formats
2666 (``CellStyleList._to_array``); a named style format's alignment and
2667 protection to theirs (``NamedStyle.bind``, which adds an absent one
2668 as the default already listed)."""
2669 if container in entry_classes:
2670 lists[container].add(entry_classes[container].from_tree(element))
2671 return
2672 from lxml import etree
2673 from openpyxl.styles.cell_style import CellStyle
2675 if etree.QName(element).localname != "xf":
2676 return
2678 xf = CellStyle.from_tree(element)
2679 alignment = (
2680 alignments.add(xf.alignment) if xf.alignment is not None else None
2681 )
2682 protection = (
2683 protections.add(xf.protection) if xf.protection is not None else None
2684 )
2685 if container == "cellXfs":
2686 style = xf.to_array()
2687 if alignment is not None:
2688 style.alignmentId = alignment
2689 if protection is not None:
2690 style.protectionId = protection
2691 cell_styles.add(style)
2694def _xlsx_format_units(code: str) -> int:
2695 """Units of ``MAX_XLSX_NUMBER_FORMAT_WORK`` one date/timedelta test
2696 of format *code* costs openpyxl."""
2697 return max(len(code), 1) * (
2698 XLSX_NUMBER_FORMAT_SCAN_WEIGHT + code.count("[")
2699 )
2702def _require_skeleton_size(
2703 archive: zipfile.ZipFile,
2704 name: str,
2705 scanned: set[tuple[str, str]],
2706 families: tuple[str, ...] = (_LXML,),
2707 *,
2708 workbook: bool = False,
2709) -> None:
2710 """Hold *name* to ``MAX_XLSX_SKELETON_PART_BYTES``, scan it with
2711 *families* (``_check_xlsx_namespaces``, which records it in
2712 *scanned*) and refuse it if
2713 it declares a DTD (refused by the scan, in its bytes) or has a
2714 comment or processing instruction below its root's children
2715 (``_refuse_xlsx_nested_node``), found by one lxml parse of the part
2716 before openpyxl parses it, walking only its comments and processing
2717 instructions. The *workbook* part's lists and elements are then held
2718 as ``_check_xlsx_workbook_tree`` holds them."""
2719 from lxml import etree
2721 try:
2722 info = archive.getinfo(name)
2723 except KeyError:
2724 return # openpyxl reports the missing part itself
2725 if info.file_size > MAX_XLSX_SKELETON_PART_BYTES:
2726 raise DecompressionBombError(
2727 "spreadsheet package part is larger than "
2728 f"{MAX_XLSX_SKELETON_PART_BYTES} bytes"
2729 )
2730 _check_xlsx_namespaces(archive, {name: frozenset(families)}, scanned)
2731 with archive.open(info) as stream:
2732 for _event, element in etree.iterparse( 2732 ↛ 2740line 2732 didn't jump to line 2740
2733 _CountedXmlStream(stream),
2734 events=("start",),
2735 resolve_entities=False,
2736 no_network=True,
2737 ):
2738 _refuse_dtd(element)
2739 break
2740 root = etree.fromstring(
2741 archive.read(info),
2742 etree.XMLParser(resolve_entities=False, no_network=True),
2743 )
2744 for node in root.iter(etree.Comment, etree.ProcessingInstruction):
2745 _refuse_xlsx_nested_node(
2746 1 if node.getparent() is root else 2, "workbook skeleton"
2747 )
2748 if workbook:
2749 _check_xlsx_workbook_tree(root)
2750 del root
2753def _check_xlsx_workbook_tree(root) -> None:
2754 """Refuse a workbook part (parsed, with no comment or processing
2755 instruction below its root's children) whose root has more than
2756 ``MAX_XLSX_TOP_LEVEL_ELEMENTS`` child elements, whose lists
2757 (``_XLSX_WORKBOOK_LISTS``) hold more entries than ``_xlsx_list_cap``
2758 allows, counted over every such list, or that has an extension list
2759 of more than ``MAX_XLSX_EXTENSIONS`` entries, an element that is
2760 not a list repeating a child's local name
2761 (``_refuse_xlsx_repeated_child``) or defined names that
2762 ``_check_xlsx_defined_names`` refuses.
2764 This covers the elements openpyxl's ``WorkbookPackage.from_tree``
2765 builds objects from: the root's children, the entries of its lists
2766 and the children of those entries and of the root's other children
2767 (an extension list's extensions, whose content it does not read).
2768 Deeper elements it skips or reads as text, and it builds a defined
2769 name from each ``definedName`` of a ``definedNames`` list, its
2770 attributes, its text and its children's."""
2771 from lxml import etree
2773 totals = dict.fromkeys(_XLSX_WORKBOOK_LISTS + ("extLst",), 0)
2774 top = 0
2775 print_work = [0]
2777 def entry(element) -> None:
2778 if len(element) == 0:
2779 return
2780 seen: set = set()
2781 parent = etree.QName(element).localname
2782 for child in element: 2782 ↛ exitline 2782 didn't return from function 'entry' because the loop on line 2782 didn't complete
2783 tag = etree.QName(child).localname
2784 _refuse_xlsx_repeated_child("workbook", parent, seen, tag)
2785 if tag == "extLst":
2786 totals[tag] += len(child)
2787 _refuse_xlsx_list_entry("workbook", tag, totals[tag])
2789 for child in root.iterchildren(etree.Element):
2790 top += 1
2791 if top > MAX_XLSX_TOP_LEVEL_ELEMENTS:
2792 raise DecompressionBombError(
2793 "spreadsheet workbook part has more than "
2794 f"{MAX_XLSX_TOP_LEVEL_ELEMENTS} top-level elements"
2795 )
2796 tag = etree.QName(child).localname
2797 if tag in totals:
2798 totals[tag] += len(child)
2799 _refuse_xlsx_list_entry("workbook", tag, totals[tag])
2800 if tag != "extLst": # openpyxl does not read inside an <ext>
2801 for item in child:
2802 entry(item)
2803 elif tag == "definedNames":
2804 _check_xlsx_defined_names(child, print_work)
2805 else:
2806 entry(child)
2809def _check_xlsx_defined_names(names, print_work: list[int]) -> None:
2810 """Refuse a ``definedNames`` list of the workbook part with a defined
2811 name whose value is longer than ``MAX_XLSX_DEFINED_NAME_CHARS``, or
2812 that brings the print names' work (*print_work*, counted over every
2813 such list of the part) over ``MAX_XLSX_PRINT_NAME_WORK``.
2815 openpyxl's ``Serialisable.from_tree`` reads a defined name's value
2816 (``attr_text``) from an ``attr_text`` or ``attr-text`` attribute
2817 (it turns ``-`` in an attribute name into ``_``), replaced by the
2818 element's text when there is any, and its other fields from
2819 attributes or from child elements of their names, so every entry is
2820 read here, whatever its tag, at the longest of its attribute values,
2821 its text and its children's. A name that starts like
2822 ``_xlnm.Print_Titles`` or ``_xlnm.Print_Area`` (openpyxl matches
2823 the reserved names as a prefix) is counted at that length squared
2824 plus ``XLSX_PRINT_NAME_BASE_UNITS``, whatever its sheet."""
2825 from lxml import etree
2827 for item in names.iterchildren(etree.Element):
2828 value = max(
2829 len(text or "")
2830 for text in (
2831 item.text,
2832 *(c.text for c in item),
2833 *item.attrib.values(),
2834 )
2835 )
2836 if value > MAX_XLSX_DEFINED_NAME_CHARS:
2837 raise DecompressionBombError(
2838 "spreadsheet defined name is longer than "
2839 f"{MAX_XLSX_DEFINED_NAME_CHARS} characters"
2840 )
2841 spellings = [item.get("name")] + [
2842 c.text for c in item if etree.QName(c).localname == "name"
2843 ]
2844 if any(
2845 (spelling or "").startswith(_XLSX_PRINT_NAMES)
2846 for spelling in spellings
2847 ):
2848 print_work[0] += value * value + XLSX_PRINT_NAME_BASE_UNITS
2849 if print_work[0] > MAX_XLSX_PRINT_NAME_WORK:
2850 raise DecompressionBombError(
2851 "spreadsheet print areas and titles cost "
2852 f"{print_work[0]} units in openpyxl's load (ceiling "
2853 f"{MAX_XLSX_PRINT_NAME_WORK})"
2854 )
2857def _refuse_dtd(element) -> None:
2858 """Refuse the part *element* belongs to if it has a document type
2859 declaration, internal subset or not.
2861 A DTD is the only way to declare an entity. lxml with
2862 ``resolve_entities=False`` keeps each reference to a declared
2863 entity as an ``_Entity`` node, and with an external subset it does
2864 the same for undeclared ones (the subset is not loaded, so they
2865 are not errors); ``iterparse`` emits no event for either, so the
2866 guard's streaming counts would miss them. OPC (ECMA-376 Part 2)
2867 forbids DTDs in the parts it defines and Office writes none in the
2868 XML parts it generates.
2869 Every part the guard parses is first read through
2870 ``_CountedXmlStream``, whose ``_XmlProlog`` refuses the declaration
2871 in the bytes, before a parser reads it; this is a second line. It looks at the root's start event, by which libxml2 has
2872 read the whole internal subset (one million attribute declarations,
2873 a 17.9 MB part, held 424 MB first)."""
2874 docinfo = element.getroottree().docinfo
2875 if ( 2875 ↛ 2880line 2875 didn't jump to line 2880 because the condition on line 2875 was never true
2876 docinfo.doctype
2877 or docinfo.internalDTD is not None
2878 or docinfo.externalDTD is not None
2879 ):
2880 raise DecompressionBombError("document package part declares a DTD")
2883#: Ceiling on the comments and processing instructions outside the root
2884#: element (before or after it) of a part the guard streams with
2885#: comment events and does not otherwise count them in: the ``.xlsx``
2886#: styles part, ``.pptx`` slides, layouts and masters, and ``.docx``
2887#: ``document.xml``, headers and footers. lxml's ``iterparse``
2888#: emits the event for one before the root in time linear in the nodes
2889#: already there, so N of them cost the guard N squared: 60,000 empty
2890#: comments before a slide's root, a 420 KB part, kept the guard busy
2891#: for 16.7 s (40,000, 5.6 s; after the root they cost nothing extra).
2892#: Office writers put none there.
2893MAX_XML_OUTSIDE_ROOT_NODES = 64
2896def _note_outside_root_node(count: int) -> int:
2897 """Count one more comment or processing instruction outside a
2898 part's root element, refusing the part past
2899 ``MAX_XML_OUTSIDE_ROOT_NODES``; returns the new count."""
2900 count += 1
2901 if count > MAX_XML_OUTSIDE_ROOT_NODES:
2902 raise DecompressionBombError(
2903 "document package part has more than "
2904 f"{MAX_XML_OUTSIDE_ROOT_NODES} comments or processing "
2905 "instructions outside its root element"
2906 )
2907 return count
2910# --- Reference fan-out in .pptx / .docx -----------------------------------
2911#
2912# python-pptx and python-docx load each package part once, but the text
2913# extractors walk *references*: unstructured's ``partition_pptx`` emits a
2914# slide once per ``<p:sldId>`` that resolves to it, and ``partition_docx``
2915# emits a header/footer part once per section that references it, after
2916# python-docx has located each section's content with XPath over the
2917# body-level elements and over every body-level paragraph's children
2918# (so N sections cost N passes over the body; see
2919# ``MAX_DOCX_SECTION_BLOCK_WORK`` and ``MAX_DOCX_SECTION_WORK``). One
2920# bounded part referenced N times is therefore N times the work and the
2921# extracted text, which no per-entry ceiling bounds. The same holds per
2922# paragraph: unstructured looks up each non-empty paragraph's style
2923# about four times (each a scan of the styles part and of the document
2924# part's relationships), reads each run's text about five times, and
2925# splits a paragraph at each rendered page break. python-docx's and
2926# unstructured's cost is not modelled exactly: the section count and
2927# the parts read per section or per paragraph are capped
2928# (``MAX_DOCX_SECTIONS``, ``MAX_PART_RELATIONSHIPS``,
2929# ``MAX_DOCX_SETTINGS_CHILDREN``, ``MAX_DOCX_STYLES``), and the work
2930# that grows with the body is counted as products of those counts,
2931# each priced at the slowest layout measured. The checks below parse in
2932# full only package skeleton parts (relationships, ``presentation.xml``,
2933# the ``.docx`` settings and styles parts), each held to
2934# ``MAX_OPC_SKELETON_PART_BYTES``; ``document.xml`` is not a skeleton
2935# part: it is streamed once without building a tree, and is bounded
2936# only by the parsed-part ceiling (``MAX_ENTRY_BYTES``, 128 MiB).
2938#: Ceiling for the skeleton parts the .pptx/.docx fan-out checks parse
2939#: in full (package and part relationships, ``presentation.xml``, the
2940#: ``.docx`` settings and styles parts). Real ones are kilobytes (a
2941#: styles part with hundreds of styles is a few hundred KB); 10,000
2942#: slides list in well under 1 MiB.
2943MAX_OPC_SKELETON_PART_BYTES = 16 * 1024 * 1024
2945#: Ceiling on the entries (children of the root element) of every
2946#: relationships part (``*.rels`` member) of a ``.docx`` or ``.pptx``
2947#: package, the parts the guard reads (the package's, the main
2948#: document's and the presentation's) and the others alike: the
2949#: libraries parse the relationships part of every part they reach
2950#: (see ``MAX_PACKAGE_RELATIONSHIPS``). python-docx finds the settings
2951#: and styles parts by scanning every relationship of the document part
2952#: in Python (uncached): unstructured does that twice per section for
2953#: the settings part, which this cap keeps small (10,000 relationships
2954#: under ``MAX_DOCX_SECTIONS`` sections measured about 3 s), and about
2955#: four times per non-empty paragraph for the styles part, which
2956#: ``MAX_DOCX_STYLE_WORK`` counts. Real documents have tens to a few
2957#: hundred relationships (one per image, hyperlink and header/footer
2958#: part, plus about a dozen fixed parts), so a document with thousands
2959#: of images or external hyperlinks still fits.
2960MAX_PART_RELATIONSHIPS = 10_000
2962#: Ceiling on the attributes of any element of a relationships part.
2963#: The OPC schema (ECMA-376 Part 2) gives ``Relationship`` four
2964#: (``Id``, ``Type``, ``Target``, ``TargetMode``) and ``Relationships``
2965#: none; namespace declarations are not attributes. Checked at each
2966#: element's start, so a part whose first entry carries hundreds of
2967#: thousands of attributes is refused after that entry rather than
2968#: read to its end.
2969MAX_RELS_ELEMENT_ATTRIBUTES = 64
2971#: Ceiling on the relationships of all the relationships parts of a
2972#: ``.docx`` or ``.pptx`` package together. python-docx and python-pptx
2973#: load every part reachable through relationships and parse each
2974#: one's relationships part, about 5-8 us per relationship (a million,
2975#: in 100 parts of 10,000 external hyperlinks each, a 5.9 MB file, took
2976#: 5.2 s to open in python-docx and 8.1 s in python-pptx), so 50,000
2977#: cost well under a second. Real documents have tens to hundreds; one
2978#: with 10,000 images and as many hyperlinks has about 20,000.
2979MAX_PACKAGE_RELATIONSHIPS = 50_000
2981#: Ceiling on python-docx's relationship walks when it opens a package:
2982#: its ``PackageReader._walk_phys_parts`` keeps the parts it has visited
2983#: in a list and, for every internal relationship of every part it
2984#: reaches, scans that list (``partname in visited_partnames``), and
2985#: ``OpcPackage.iter_rels`` (run on load to gather the image parts)
2986#: does the same with its own list. The guard counts the internal
2987#: relationships (``TargetMode`` other than ``External``) of every
2988#: relationships part times the archive's members (an upper bound on
2989#: the parts visited: each must exist), each member counted as one step
2990#: plus one per ``DOCX_RELATIONSHIP_WALK_NAME_BYTES_PER_STEP`` bytes of
2991#: its name as CPython stores it (comparing names of equal length reads
2992#: them: one, two or four bytes per character, by the name's widest
2993#: code point; ``_str_width``). A step measured 17-25 ns: 9,000 parts
2994#: and one part with 200,000 relationships to the last of them, a
2995#: 1.66 MB file with every relationships part the guard read under
2996#: ``MAX_PART_RELATIONSHIPS``, took 41 s to open before the change, and
2997#: 2,000 parts with 20,000, 1.0 s; 9,000 parts with 19,000 internal
2998#: relationships, counted at 0.91 of this ceiling, open in 3.0 s. Names
2999#: of four-byte characters cost two to four times as much per character
3000#: as ASCII ones and were counted per character before: 2,400 parts
3001#: named by 450 emoji and 10,114 internal relationships open in 2.1 s
3002#: and are counted at 1.01 of this ceiling (0.34 per character), and
3003#: 2,000 parts named by 2,000 emoji and 10,014 relationships, counted
3004#: at 3.3 (0.89 per character), in 9.5 s, so opening at the ceiling
3005#: costs about 2-5 s whatever the names. A document with 10,000 images
3006#: (one part and one relationship each) counts about 110 million.
3007MAX_DOCX_RELATIONSHIP_WALK = 200_000_000
3008DOCX_RELATIONSHIP_WALK_NAME_BYTES_PER_STEP = 256
3010#: Ceiling on the ``w:sectPr`` elements anywhere in ``document.xml``
3011#: (python-docx enumerates those in body-level paragraph properties and
3012#: at body level as sections; any other is counted too). python-docx and
3013#: unstructured do work once per section that this guard does not model
3014#: exactly: each section re-evaluates the ``_sectPrs`` XPath (whose
3015#: union merge compares the two kinds of ``w:sectPr`` pairwise and whose
3016#: returned nodes cost about 100 ns each), and unstructured reads the
3017#: settings part's ``evenAndOddHeaders`` twice (a scan of the document
3018#: part's relationships and a search of the settings root's children).
3019#: This cap, with ``MAX_PART_RELATIONSHIPS`` and
3020#: ``MAX_DOCX_SETTINGS_CHILDREN``, bounds each of those factors instead:
3021#: 1,000 sections packed into one paragraph's properties, or split
3022#: between paragraph and body level, partition in under 2 s. Word writes
3023#: one ``w:sectPr`` per section break plus the body's own; real
3024#: documents rarely have more than a few hundred (a mail merge to
3025#: "individual documents" writes one per record). Each section's
3026#: paragraphs also count toward ``MAX_DOCX_SECTION_BLOCK_WORK`` and the
3027#: shared budget: measured on documents python-docx writes (where each
3028#: section break is a paragraph of its own), more than about 580
3029#: one-paragraph sections, or about 180 ten-paragraph ones, are
3030#: refused.
3031MAX_DOCX_SECTIONS = 1_000
3033#: Ceiling on the children of the settings part's root element. The
3034#: part is the one the document part's settings relationship names, as
3035#: python-docx resolves it, and unstructured searches its root's
3036#: children twice per section. Word's settings.xml has well under 200.
3037MAX_DOCX_SETTINGS_CHILDREN = 5_000
3039#: Ceiling on the quadratic part of python-docx's per-section content
3040#: lookup: the sum, over sections, of the body-level paragraphs and
3041#: tables (``w:p``/``w:tbl``) from the start of the body to the
3042#: section's end, times the child nodes of the body (elements, comments,
3043#: processing instructions and text). python-docx 1.2 finds a section's
3044#: blocks with ``preceding-sibling::*[self::w:p | self::w:tbl]`` from the
3045#: section's end, which returns that section's blocks and every earlier
3046#: one, and libxml2 (2.14 measured) sorts that reverse-axis result into
3047#: document order with comparisons that each walk sibling links towards
3048#: the end of the body. One evaluation therefore costs up to (blocks
3049#: returned) x (body nodes) steps; it runs once per section. A step
3050#: costs about 5 ns while the body fits in the CPU caches, but a large
3051#: body is walked from memory, and the cost then grows with what each
3052#: node occupies: measured end to end in ``partition_docx`` behind 1,240
3053#: paragraphs, about 16 ns per step for bare elements, ~48 ns with three
3054#: attributes, ~70-100 ns with 8-40 attributes or twenty child elements,
3055#: and ~80 ns per counted step for three-attribute elements separated by
3056#: text. The ceiling prices every step at that slowest layout, ~100 ns,
3057#: so this fan-out at the ceiling measured about 45 s (1,240 paragraphs
3058#: before 348,000 twenty-attribute bookmarks, at 96% of it: 40 s; before
3059#: the change, 1,240 paragraphs and 2,000,000 three-attribute bookmarks,
3060#: a 234 KiB file, were accepted and took 121 s). Bodies of plain
3061#: paragraphs that python-docx would walk at a few ns per step are
3062#: priced the same. On its own the ceiling would admit a single section
3063#: of about 21,000 paragraphs and tables; with the shared budget (the
3064#: style lookup costs about as much per paragraph) the measured limits
3065#: are about half that (see "Known gaps" in the module docstring).
3066#: Text between body-level elements is counted whether or not it is
3067#: whitespace (python-docx's parser drops most whitespace-only text, but
3068#: not all), so a pretty-printed ``document.xml`` counts up to twice its
3069#: nodes.
3070MAX_DOCX_SECTION_BLOCK_WORK = 450_000_000
3072#: Ceiling on the linear part of python-docx's per-section work, in
3073#: units of one body node per section: the section count times (the
3074#: body's child nodes, which each section's sibling walk and
3075#: ``count()`` visit, plus the nodes the ``_sectPrs`` XPath visits
3076#: divided by ``DOCX_SECTION_XPATH_NODES_PER_UNIT``, plus the node pairs
3077#: its union compares divided by ``DOCX_SECTION_UNION_PAIRS_PER_UNIT``).
3078#: The ``_sectPrs`` XPath is
3079#: ``/w:document/w:body/w:p/w:pPr/w:sectPr | /w:document/w:body/w:sectPr``;
3080#: its child steps visit every child node (elements, comments,
3081#: processing instructions, text) of the document node, of
3082#: ``w:document``, of each ``w:body``, of each body-level ``w:p`` and of
3083#: each such paragraph's ``w:pPr``, at 12-33 ns per visit, and libxml2
3084#: merges its two branches by comparing every paragraph-level
3085#: ``w:sectPr`` with every body-level one (1-4 ns per pair). The weights
3086#: are generous rather than exact: at this ceiling 1,000 body-level
3087#: sections over ~9,700 body nodes took 1-2 s in ``partition_docx``.
3088MAX_DOCX_SECTION_WORK = 10_000_000
3090#: Section-XPath node visits that count as one unit of
3091#: ``MAX_DOCX_SECTION_WORK`` (one body-level element per section).
3092DOCX_SECTION_XPATH_NODES_PER_UNIT = 64
3094#: Node pairs compared by the section XPath's union (per evaluation)
3095#: that count as one unit of ``MAX_DOCX_SECTION_WORK``.
3096DOCX_SECTION_UNION_PAIRS_PER_UNIT = 1_000
3098#: Ceiling on header/footer bytes as extracted: each section's
3099#: header/footer reference costs one text read of the referenced part,
3100#: so the referenced part's size is counted once per reference. A read
3101#: costs up to about 3.5 us per byte (empty runs, empty paragraphs or
3102#: empty table cells: 20 references to a 1.2 MB header of empty runs
3103#: took ~73 s), so the sum is held to 12 MiB, about 45 s (39-40 s
3104#: measured for this fan-out at the ceiling). Real documents reference kilobyte-sized
3105#: headers from a handful of sections; a mail merge of 1,000 records
3106#: whose header and footer total 12 KB would be refused. Each read also
3107#: evaluates the part's node-set unions, which are not linear in its
3108#: size (see ``DOCX_UNION_STEPS_PER_HEADER_BYTE``): a header holding a
3109#: table before 8,000 empty paragraphs and 40,000 empty elements,
3110#: referenced by ten sections, a 37 KB file, took 35 s and was counted
3111#: at 0.23 of this ceiling before.
3112MAX_DOCX_HEADER_FOOTER_BYTES = 12 * 1024 * 1024
3114#: Ceiling on the children of the styles part's root element (the part
3115#: the document part's styles relationship names, as python-docx
3116#: resolves it). Word writes the styles a document uses plus its
3117#: defaults, typically tens to a few hundred ``w:style`` elements;
3118#: python-docx's own template has 164.
3119MAX_DOCX_STYLES = 5_000
3121#: Ceiling on unstructured's style lookups: the paragraphs that can have
3122#: text (body-level ``w:p`` with a ``w:r`` or ``w:hyperlink`` child, plus
3123#: one per ``w:lastRenderedPageBreak`` in them, since each page break
3124#: splits off a fragment that is looked up again) times (the styles
3125#: root's children, plus their attributes divided by
3126#: ``DOCX_STYLE_ATTRIBUTES_PER_ENTRY``, plus the document part's
3127#: relationships divided by ``DOCX_RELATIONSHIPS_PER_STYLE_ENTRY``).
3128#: unstructured reads ``paragraph.style`` about four times per such
3129#: paragraph; each read scans every relationship of the document part in
3130#: Python (``part_related_by``) and, for a paragraph whose style is
3131#: unset or does not resolve, loops over every ``w:style`` in Python
3132#: (``default_for``), reading two attributes of each. Measured at about
3133#: 15 us per unit (50 paragraphs beside 300,000 styles, an 805 KB file,
3134#: took 132 s before the change), so this fan-out at the ceiling is
3135#: about 45 s: 597 paragraphs beside 5,000 styles took 40 s, and 5,664
3136#: paragraphs beside 10,000 relationships, at 90% of it, 24 s. On its
3137#: own it would admit about 18,000 non-empty paragraphs under
3138#: python-docx's template (166 entries), fewer with more styles; with
3139#: the shared budget the measured limits are lower (see "Known gaps").
3140MAX_DOCX_STYLE_WORK = 3_000_000
3142#: Document-part relationships that count as one styles-part entry in
3143#: ``MAX_DOCX_STYLE_WORK`` (a relationship costs about 0.1 us per read,
3144#: a style about 3.5 us).
3145DOCX_RELATIONSHIPS_PER_STYLE_ENTRY = 32
3147#: Attributes of the styles root's children that count as one entry in
3148#: ``MAX_DOCX_STYLE_WORK``: libxml2 finds an attribute by scanning the
3149#: element's attributes, so styles of 50 or 250 attributes cost 1.4 or
3150#: 3.1 times as much per read as plain ones.
3151DOCX_STYLE_ATTRIBUTES_PER_ENTRY = 32
3153#: Ceiling on the sibling sorting of unstructured's page-break splits:
3154#: the sum over body-level paragraphs of K * (K + 3) * S**2, K being the
3155#: paragraph's ``w:lastRenderedPageBreak`` elements and S the largest
3156#: child-node count of the paragraph or of any element in it.
3157#: unstructured splits a paragraph at its first rendered page break and
3158#: recurses into the rest; each split evaluates python-docx XPath with
3159#: reverse sibling axes (``precedes_all_content`` once per break left in
3160#: the first run, and the fragment builders), each sorted at up to S**2
3161#: steps. Measured at up to ~5 ns per unit (one break after 55,000
3162#: elements of one run took 34-61 s; one run with 600 breaks, a 37 KB
3163#: file, ran for over 180 s before the change), so this fan-out at the
3164#: ceiling is about 45 s. Word writes about one rendered page break per
3165#: page, in paragraphs of tens to hundreds of runs.
3166MAX_DOCX_PAGE_BREAK_WORK = 9_000_000_000
3168#: Ceiling on the ``w:lastRenderedPageBreak`` elements anywhere in
3169#: ``document.xml``. unstructured asks once per document whether it has
3170#: any, with a four-branch union over the whole body
3171#: (``w:body/w:p/w:r``, ``w:body/w:p/w:hyperlink/w:r`` and the same two
3172#: under ``w:body/w:tbl/w:tr/w:tc``), and libxml2 merges each branch by
3173#: comparing every node with every node already merged, then sorts the
3174#: result with comparisons that walk the siblings after the breaks:
3175#: 20,000 breaks in one table cell after 20,000 in body paragraphs took
3176#: 0.8 s, quadrupling per doubling, and 10,000 breaks in each of the two
3177#: cell branches with 800-1,200 empty elements between consecutive ones
3178#: 2.3-2.5 s (~25 ns per pair), while a break in a cell costs the other
3179#: ceilings about one unit of ``MAX_DOCX_RUN_WORK``. Word writes one
3180#: rendered page break per page in each flow of text the page boundary
3181#: cuts (the body, and each cell of a table row the boundary crosses),
3182#: so a 1,000-page document of ten-column tables has about 10,000 and a
3183#: book of 20,000 pages this many. This ceiling bounds the merges, not
3184#: the sort: once a later branch's break precedes an earlier branch's,
3185#: the sort walks the shared parent's children after the breaks, so
3186#: filler there costs per break (1,000 breaks of each hyperlink and
3187#: plain branch in table rows before 160,000 empty table children, a
3188#: 42 KB file, took 2.2-2.6 s per evaluation, 640,000 of them 10.5 s,
3189#: and 10,000 rows of each branch alternating with empty spacer rows
3190#: 3.8-7.6 s). The merges' pairs and the sort's steps are both priced
3191#: in ``MAX_DOCX_RUN_WORK`` (``DOCX_PAGE_BREAK_UNION_PAIRS_PER_UNIT``,
3192#: ``DOCX_PAGE_BREAK_SORT_STEPS_PER_UNIT``): those three shapes were
3193#: counted at 0.02, 0.02 and 0.36 of it before, and are now counted at
3194#: 1.24, 4.85 and 3.06 of it and refused; the sort's price covers
3195#: siblings that lie far apart in memory and libxml2's binary-insertion
3196#: runs, which plain filler and 1,000 breaks per branch do not reach,
3197#: so the union at that ceiling costs about 30 s. Before the change, 96,000
3198#: breaks in each of the two cell branches, about 0.9 of
3199#: ``MAX_DOCX_RUN_WORK``, were accepted and would take ~18 s
3200#: (extrapolated: a 45 KB file with 20,000 in each took 0.77 s and was
3201#: counted at 0.19 of it). ``w:br w:type="page"`` is
3202#: not part of this union (it is one of a run's text branches, see
3203#: ``DOCX_UNION_STEPS_PER_RUN_WORK_UNIT``) and is not counted here.
3204MAX_DOCX_PAGE_BREAKS = 20_000
3206#: Node pairs compared by that union's merges (the sum, over pairs of
3207#: branches, of the product of their break counts) that count as one
3208#: unit of ``MAX_DOCX_RUN_WORK``: 2-3 ns per pair for the merges and up
3209#: to ~25 ns with the sort measured, priced at 30 ns.
3210DOCX_PAGE_BREAK_UNION_PAIRS_PER_UNIT = 1_000
3212#: Steps of that union's sort that count as one unit of
3213#: ``MAX_DOCX_RUN_WORK``, a step priced at 150 ns. With two or more
3214#: non-empty branches, the merged set is out of document order as soon
3215#: as a later branch's break precedes an earlier branch's, and libxml2
3216#: then compares breaks under a common ancestor by walking that
3217#: ancestor's children from one towards the other, to the end of them
3218#: when the order is reversed. The guard counts, for every element, the
3219#: union's breaks below it, B, times its child nodes (text included),
3220#: weighted by ``B + DOCX_PAGE_BREAK_SORT_INSERTION_WALKS *
3221#: min(B, DOCX_PAGE_BREAK_SORT_INSERTION_RUN)``, summed, and only when
3222#: at least two branches are non-empty (one branch is already in
3223#: document order). The weight covers libxml2's timsort: below 64 nodes,
3224#: and for each run of the merged set shorter than its minimum run
3225#: (32-64 nodes; the merged set is at most four ascending runs, one per
3226#: branch), it sorts by binary insertion, about log2(n) comparisons, so
3227#: up to six walks, per node: 62 breaks of one branch before one of
3228#: another walked the filler after them 12 times per break, against
3229#: about 2.5 times for 2,000 breaks. Each walk step is a dependent load
3230#: of the next sibling, so its cost is set by how far apart siblings
3231#: lie in memory, not by what they are: 3-9 ns for siblings allocated
3232#: back to back, and 35-45 ns, a cache miss each, once each carries 16
3233#: or more attributes, a few descendants or text (64 and 256 attributes
3234#: cost no more). Measured per weighted step, the worst shapes found
3235#: (62 breaks of one branch and one of another, or 31 of each of four
3236#: branches, before 150,000-200,000 filler elements with 16-64
3237#: attributes each; 1,000 breaks of one branch and one of another before
3238#: 100,000 filler elements with 64 attributes) cost 70-95 ns, so the
3239#: sort at the ceiling costs about 30 s. Priced at 30 ns per unweighted
3240#: step before, 1,000 breaks of each of two branches in table rows
3241#: before 600,000 filler elements with 16 empty attributes each, a
3242#: 386 KB file, were accepted at 0.83 of ``MAX_DOCX_RUN_WORK`` and took
3243#: 50 s for one evaluation of the union.
3244DOCX_PAGE_BREAK_SORT_STEPS_PER_UNIT = 200
3246#: Extra walks per break of the binary-insertion runs of that union's
3247#: sort, and the breaks they apply to (see
3248#: ``DOCX_PAGE_BREAK_SORT_STEPS_PER_UNIT``).
3249DOCX_PAGE_BREAK_SORT_INSERTION_WALKS = 4
3250DOCX_PAGE_BREAK_SORT_INSERTION_RUN = 64
3252#: Ceiling on unstructured's text extraction from body-level paragraphs
3253#: and tables: the sum over them of (K + 1) * (``DOCX_RUN_WORK_ITEM_WEIGHT``
3254#: * (``w:r``, ``w:tr`` and ``w:tc`` elements) + child nodes of the
3255#: ``w:r`` elements) + K * (descendant nodes) //
3256#: ``DOCX_FRAGMENT_NODES_PER_RUN_WORK_UNIT``, K being a paragraph's
3257#: rendered page breaks (0 for a table). unstructured reads each
3258#: fragment's text about five times, and each read of a run is an XPath
3259#: evaluation (~25 us) plus a Python loop over the run's children;
3260#: tables cost about as much per row and cell (cells counted as
3261#: ``_DocxSectionStats.run_work`` describes, with python-docx's steps up
3262#: vertical merges priced by ``DOCX_MERGE_STEP_WEIGHT``); and each
3263#: page-break split deep-copies the paragraph. Measured at up to ~30 us
3264#: per unit (1.8 million tabs in one run; 120 paragraphs of 2,479 one-character
3265#: runs, a 76 KB file, took 40 s, and the cost was bounded only by the
3266#: 128 MiB part ceiling before the change), so this fan-out at the
3267#: ceiling is about 45 s. On its own it would admit about 50,000
3268#: paragraphs of five runs; with the shared budget, far fewer (see
3269#: "Known gaps").
3270MAX_DOCX_RUN_WORK = 1_500_000
3272#: Units of ``MAX_DOCX_RUN_WORK`` per run, table row or table cell.
3273DOCX_RUN_WORK_ITEM_WEIGHT = 5
3275#: Descendant nodes of a split paragraph that count as one unit of
3276#: ``MAX_DOCX_RUN_WORK`` per page break (the deep copy and the walks of
3277#: each fragment: ~650 ns per node and break measured).
3278DOCX_FRAGMENT_NODES_PER_RUN_WORK_UNIT = 32
3280#: Ceiling on every ``w:gridSpan``, ``w:gridBefore`` and ``w:gridAfter``
3281#: value in ``document.xml``. python-docx's ``_Row.cells`` yields a cell
3282#: once per grid column it spans, and unstructured re-reads the cell's
3283#: text for each yield (and pads a row with one empty string per
3284#: ``gridBefore``/``gridAfter`` column), so the value is a multiplier
3285#: on a cell's work and on the text it retains: a 950-byte file with
3286#: one cell spanning 400,000 columns took 53 s, and at 2**31 - 1
3287#: python-docx would build a tuple of about 17 GB. Word's tables are
3288#: at most 63 columns wide, and the values of any writer are bounded
3289#: by the table grid; 1,000 leaves 16 times Word's width. A value
3290#: python-docx cannot read as an integer (``int()`` of the attribute,
3291#: as python-docx converts it) is refused too, as is a missing one or
3292#: one of more than ``_MAX_DOCX_GRID_VALUE_CHARS`` characters. Negative
3293#: values are accepted (they span and pad nothing).
3294MAX_DOCX_GRID_SPAN = 1_000
3296#: Longest ``w:val`` of a grid element the guard parses.
3297_MAX_DOCX_GRID_VALUE_CHARS = 32
3299#: Most attributes any one element of ``document.xml`` may carry. lxml
3300#: reads each attribute value by scanning the element's attribute list
3301#: again (``attrib.values()``, ``items()``), so reading them all is
3302#: quadratic in their number: 10,000 attributes took 0.35 s, 40,000
3303#: 11 s, and one paragraph of 50,000 empty attributes, a 145 KB file,
3304#: kept this guard itself busy for 40 s before. Real writers put at
3305#: most a few dozen on an element (Word's ``w:p`` carries up to seven
3306#: revision and paragraph ids, ``wp:anchor`` eleven; the most found in
3307#: Word, LibreOffice and python-docx output was twelve, and VML shapes
3308#: allow a few dozen), so 256 is refused outright, checked by counting
3309#: (linear) before any attribute is read. lxml has built the element by
3310#: then; one of more than ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` attributes is
3311#: refused before lxml receives it (``_CountedXmlStream``).
3312MAX_DOCX_ELEMENT_ATTRIBUTES = 256
3314#: Most namespace declarations in scope at any element of
3315#: ``document.xml`` (its own and its ancestors'). libxml2 resolves the
3316#: namespace of every element and attribute it copies by walking the
3317#: declarations in scope, so a paragraph whose elements use N
3318#: declarations costs N squared per copy, and python-docx copies a
3319#: paragraph twice per rendered page break: 8,000 declarations on the
3320#: ``w:document`` element used by 8,000 elements took 0.6 s per copy,
3321#: and the cost quadruples per doubling. Word declares about 35 on
3322#: ``w:document`` and drawings add a few each; 256 is refused outright.
3323MAX_DOCX_NAMESPACES_IN_SCOPE = 256
3325#: Most namespace declarations inside one body-level paragraph of
3326#: ``document.xml`` (on the ``w:p`` and on everything in it). lxml's
3327#: ``remove()``, which python-docx calls on a copy of the paragraph for
3328#: every rendered page break, re-homes the removed subtree's namespace
3329#: declarations through a cache it searches linearly, so its cost grows
3330#: with the square of the declarations in that subtree: 5,000 one-
3331#: declaration elements took 0.007 s per removal, 40,000 0.4-0.8 s, and
3332#: six page breaks before 40,000 of them, a 142 KB file, took 5 s to
3333#: partition and were accepted before (at the copied-bytes ceiling a
3334#: paragraph of 480,000 would cost minutes). At this cap a removal
3335#: costs well under a millisecond (a paragraph at the cap with 140 page
3336#: breaks partitioned in 1.7 s, against 1.65 s without the
3337#: declarations). Word declares its namespaces on ``w:document`` and up
3338#: to five more per inline picture (python-docx two), so 1,024 admits
3339#: paragraphs of 200 pictures. Tables are not split, so they are not
3340#: capped.
3341MAX_DOCX_PARAGRAPH_NAMESPACES = 1_024
3343#: Longest namespace URI, and longest prefix, a declaration in
3344#: ``document.xml`` may have, in characters; checked when the
3345#: declaration is announced, before any element name is read. lxml
3346#: builds every element and attribute name it hands back as
3347#: ``{URI}local``, so one long URI declared once and used through a
3348#: short prefix is copied again for every name read under it: a single
3349#: 8 MB URI used by 3,000 elements, a 44 KB file, ran this guard out of
3350#: a 3 GB address space within 3 s (a 1 MB one took 3.5 s), and a 1 MB
3351#: URI on 1,024 attribute names grew it by ~0.5 GB. With the cap, the
3352#: guard reads each name a bounded number of times, so a name costs it
3353#: at most about 1 KB per read (100,000 elements named through a URI
3354#: at the cap took about a third longer than through Word's own). The
3355#: namespaces Office writers use are well under 100
3356#: characters (the longest in Word's output,
3357#: ``http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing``,
3358#: has 71), and their prefixes under ten.
3359MAX_DOCX_NAMESPACE_URI_CHARS = 1_024
3360MAX_DOCX_NAMESPACE_PREFIX_CHARS = 64
3362#: Most ``w:tc`` children of one ``w:tr``, and most grid columns one row
3363#: may cover (its ``gridBefore``, the spans of its cells and its
3364#: ``gridAfter``). python-docx resolves a cell that continues a
3365#: vertical merge (``row.cells``) by computing its grid offset, which
3366#: reads the span of every earlier cell of its row, and then walking
3367#: the row above to that offset, recursively up the merge; unstructured
3368#: reads ``row.cells`` twice per table. Each such step therefore costs
3369#: time proportional to the row's width, and a merged row costs its
3370#: width squared: two rows of 1,000 cells, the second all continuing, a
3371#: 37 KB file, took 6-7 s per read of the cells (4,000 cells, minutes).
3372#: The steps are priced in ``MAX_DOCX_RUN_WORK`` (see
3373#: ``DOCX_MERGE_STEP_WEIGHT``); this cap keeps each step's price, which
3374#: the guard bounds linearly in the row's width, inside the range it was
3375#: measured over (past a few hundred cells a step grows faster than
3376#: linearly; a merged row of 256 cells: ~0.4 s per read, 0.84 s to
3377#: partition). Word's tables are at most
3378#: 63 columns wide, so 256 cells is four times that; 2,048 columns
3379#: admits a row of a cell at the ``MAX_DOCX_GRID_SPAN`` cap with that
3380#: much padding on both sides.
3381MAX_DOCX_ROW_CELLS = 256
3382MAX_DOCX_ROW_GRID_COLUMNS = 2_048
3384#: Units of ``MAX_DOCX_RUN_WORK`` per step python-docx takes to resolve
3385#: a cell continuing a vertical merge (``_tc_above``: two XPath
3386#: evaluations and the grid offset computations, about 0.1 ms per read
3387#: of ``row.cells`` for a one-column table), plus one per
3388#: ``DOCX_MERGE_CELLS_PER_UNIT`` cells whose span it reads (the cells
3389#: before the continuing cell in its row, and those before the cell it
3390#: lands on in the row above, or all of them if there is none: 2-4 us
3391#: each per read) and one per ``DOCX_MERGE_NODES_PER_UNIT`` nodes it
3392#: walks: the child and descendant nodes of the row before the cell and
3393#: of the row above (which those reads and the row above's cell list
3394#: walk); the row's children up to its first ``w:trPr`` (all of them if
3395#: it has none; capped by ``MAX_DOCX_ROW_OTHER_CHILDREN``) and the nodes
3396#: in that ``w:trPr`` (``grid_before``'s ``find`` calls, wherever the
3397#: ``w:trPr`` sits); and the table's children from the row above
3398#: through the row (``_tr_above``; capped by
3399#: ``MAX_DOCX_TABLE_ROW_GAP``). A cell whose merge goes up D rows takes D
3400#: steps, each priced at its own rows, so one column merged down R rows
3401#: costs R squared over two steps. The prices cover both of
3402#: unstructured's reads: end to end in ``partition_docx``, merged
3403#: tables of 1-256 columns and 2-500 rows (and rows with 256 other
3404#: children and 256 more between each two, or cells whose ``w:tcPr``
3405#: holds 2,000 children), at 0.84-0.96 of the ceiling, took 1-21 us per
3406#: unit counted, under the ~30 us per unit the ceiling assumes; 64
3407#: columns merged down 26 rows, at 0.96 of the ceiling, took 22 s.
3408DOCX_MERGE_STEP_WEIGHT = 10
3409DOCX_MERGE_CELLS_PER_UNIT = 2
3410DOCX_MERGE_NODES_PER_UNIT = 16
3412#: Most children of a table row python-docx reads (a direct ``w:tr``
3413#: child of a ``w:tbl``) that are not ``w:tc`` elements, and most
3414#: children of a ``w:tbl`` other than ``w:tr`` elements before or
3415#: between two of its ``w:tr`` children (``w:tblPr`` and ``w:tblGrid``
3416#: included); elements, comments and processing instructions count
3417#: (the text between them is at most one node more each). Every step
3418#: up a vertical merge reads ``grid_before`` of the continuing cell's
3419#: row, an lxml ``find`` over all the row's children when it has no
3420#: ``w:trPr``, and evaluates ``_tr_above``
3421#: (``ancestor::w:tr[1]/preceding-sibling::w:tr[1]``), which visits
3422#: every sibling between that row and the one above. The siblings are
3423#: walked, not their content: a row padded with 200,000 empty elements
3424#: after its cells, a 1.6 KB file, took 7.6 s per read of the cells,
3425#: and 26 merged rows of 64 cells with 200,000 between each pair, an
3426#: 81 KB file, 58 s per read (unstructured reads twice), at 0.09 and
3427#: 0.90 of the ceiling before; 20,000 wrapped in one ``w:sdt`` per
3428#: row cost nothing. In a table
3429#: Word, LibreOffice and python-docx write, a row's other children are
3430#: its ``w:trPr`` and ``w:tblPrEx`` and the children between rows are
3431#: none (the ``w:tblPr`` and ``w:tblGrid`` before the first row); the
3432#: schema also allows bookmark, comment, permission and revision range
3433#: markers, ``w:proofErr``, and ``w:sdt``/``w:customXml`` wrappers
3434#: there, one element per wrapped cell or group of rows. 256, the
3435#: ``MAX_DOCX_ROW_CELLS`` cap, admits Word's widest row (63 columns)
3436#: with every cell wrapped and room for markers; a step at both caps
3437#: takes ~0.12 ms per read instead of ~0.10 ms, and the siblings are
3438#: priced in it (see ``DOCX_MERGE_STEP_WEIGHT``). Children after a
3439#: table's last row, which no step walks, are not capped (a table may
3440#: hold all its rows in ``w:sdt`` wrappers).
3441MAX_DOCX_ROW_OTHER_CHILDREN = 256
3442MAX_DOCX_TABLE_ROW_GAP = 256
3444#: Descendant nodes of a table cell that count as one unit of
3445#: ``MAX_DOCX_RUN_WORK``, before the cell's work is multiplied by its
3446#: span (or added for each cell continuing its merge).
3447#: unstructured reads a cell's content once per grid column it is
3448#: yielded for (``iter_inner_content`` and ``paragraphs`` walk its
3449#: children, ``Paragraph.text`` each paragraph's), so content other
3450#: than runs is walked span times too: a cell spanning 1,000 columns
3451#: with 60,000 empty elements after its paragraph took ~1.7 s to
3452#: partition (~28 ns per node and yield) and was counted 5,005 units.
3453#: Real cells hold tens of nodes, which count nothing. That walk is
3454#: linear only for nodes python-docx skips; the cell's paragraphs and
3455#: tables are priced as items (``DOCX_CELL_BLOCK_ITEM_WEIGHT``) and the
3456#: union that selects them by ``DOCX_UNION_STEPS_PER_RUN_WORK_UNIT``.
3457DOCX_CELL_NODES_PER_UNIT = 256
3459#: Units of ``MAX_DOCX_RUN_WORK`` per ``w:p`` or ``w:tbl`` child of a
3460#: table cell, before the cell's work is multiplied by its span:
3461#: unstructured builds a python-docx object for each of them and reads
3462#: its text (and a paragraph's runs for emphasis) once per yield, about
3463#: 20 us for an empty paragraph and 7 us for an empty table measured
3464#: (16,000 of either in one cell, 1 against 8 yields).
3465DOCX_CELL_BLOCK_ITEM_WEIGHT = 1
3467#: Node-set unions python-docx and unstructured evaluate on each read
3468#: of a cell, paragraph or run: ``./w:p | ./w:tbl`` over a cell's
3469#: children (and a header's, and, in a document with no sections, the
3470#: body's), ``w:r | w:hyperlink`` over a paragraph's, and the six
3471#: text-element branches of ``CT_R.text`` over a run's.
3472#: libxml2 (2.14 measured) merges each branch's result into the
3473#: previous ones by comparing every node with every node already
3474#: there, then sorts the merged set into document order with
3475#: comparisons that each walk sibling links towards the end of the
3476#: parent; the merged set is out of order as soon as a later branch's
3477#: node precedes an earlier branch's, so one evaluation with M nodes
3478#: selected out of N child nodes costs up to M * N steps (per branch
3479#: merged after the first): ~2 ns a step in cache, ~60 ns from memory
3480#: in isolation (one ``w:tbl`` before 10,000 twenty-attribute
3481#: paragraphs and 40,000 forty-attribute elements: 31 s for one
3482#: evaluation), and up to ~115 ns per step and read end to end in
3483#: ``partition_docx`` (1,000 ``w:t`` before a ``w:br`` and 40,000 to
3484#: 60,000 elements of 40-120 attributes in one run). A step is priced
3485#: at 120 ns, so one unit (~30 us) is 250 steps of one evaluation; the
3486#: steps of a cell, paragraph or run are counted ``DOCX_UNION_READS``
3487#: times (then multiplied, like the rest of its work, by its span, its
3488#: page breaks or its merges). At 0.82-0.86 of the ceiling under the
3489#: 100 ns first tried, such a run took 35 s, such a paragraph (a
3490#: hyperlink before 1,000 runs and 60,000 forty-attribute elements)
3491#: 21 s and such a cell (a table before 1,000 paragraphs and 60,000
3492#: forty-attribute elements) 11 s. A body-level paragraph's text is
3493#: also read with unstructured's
3494#: ``w:r | w:hyperlink | w:r/descendant::wp:inline[ancestor::w:drawing][1]//w:r``,
3495#: whose third branch selects the runs inside an inline drawing of each
3496#: of the paragraph's runs: those are counted as a third branch
3497#: (any ``w:r`` under a ``wp:inline`` under a ``w:drawing`` in one of
3498#: the paragraph's ``w:r`` children), and since they are not the
3499#: paragraph's children, and libxml2 merges each run's share of them
3500#: into the earlier shares the same way, the evaluation is counted at
3501#: M * max(M, N) steps: a run whose inline drawing holds one run, before
3502#: 10,000 plain runs (a 37 KB file), took 0.37 s per evaluation, and
3503#: before 40,000, 8.9 s; ten runs whose drawings hold 5,000 runs each,
3504#: 2.9 s (counted at 0.04 and 0.17 of the ceiling before, so that
3505#: 0.9 of it would have bought minutes). Sorting two runs of one drawing
3506#: walks that drawing's nodes after the earlier one, so such an
3507#: evaluation also counts M times the child nodes of every node inside
3508#: the paragraph's runs' inline drawings (the inlines' own included):
3509#: one run with a hyperlink after it, whose drawing holds 1,000 runs
3510#: before 100,000 empty elements (a 38 KB file), took 0.31-0.39 s per
3511#: evaluation and was counted at 0.04 of the ceiling before (1.65 now);
3512#: the costliest accepted shape measured with it, 30 runs before 1.8
3513#: million elements in one drawing (52 KB, 0.92 of the ceiling), takes
3514#: 2.3 s per evaluation (~40 ns per step). Only an evaluation with at least
3515#: two non-empty branches is counted: one branch needs no merge and is
3516#: already in order (an evaluation already in order costs only the
3517#: merge's comparisons, but is counted the same). Before the change, a 37 KB file with one
3518#: cell spanning 1,000 columns that held 16,000 empty paragraphs and
3519#: 16,000 empty tables was accepted at 0.09 of the ceiling and took
3520#: ~1.5 s per yield (~25 minutes).
3521DOCX_UNION_STEPS_PER_RUN_WORK_UNIT = 250
3523#: Evaluations counted per union of a cell, paragraph or run. Measured
3524#: per body-level paragraph fragment: five of each run's union and four
3525#: of the paragraph's (its text twice, unstructured's classification
3526#: lookup and its hyperlink walk); per cell, span + 1 of the cell's
3527#: union and of its paragraphs' and 2 * span + 1 of its runs' (one or
3528#: two per yield and one for the table's text), which six per yield
3529#: covers.
3530DOCX_UNION_READS = 6
3532#: Union steps of one header/footer read that count as one byte of
3533#: ``MAX_DOCX_HEADER_FOOTER_BYTES`` (a byte read is priced at ~3.5 us,
3534#: a step at 120 ns): each reference reads the part's root union, and
3535#: each of its cells', paragraphs' and runs' unions, once.
3536DOCX_UNION_STEPS_PER_HEADER_BYTE = 29
3538#: Attributes up to which the guard reads an element's attribute values
3539#: with ``attrib.values()``, quadratic in their number but fastest for
3540#: few (about 0.4 us for three); above it, with an ``@*`` XPath, linear
3541#: with a larger fixed cost (about 60 us for 256 against 140 us).
3542_DOCX_ATTRIBUTE_VALUES_SCAN = 64
3544#: ``gridBefore``/``gridAfter`` padding columns that count as one unit
3545#: of ``MAX_DOCX_RUN_WORK`` (one empty string yielded and one ``<td/>``
3546#: emitted each: ~0.5 us and ~20 bytes measured).
3547DOCX_GRID_PADDING_PER_RUN_WORK_UNIT = 8
3549#: Ceiling on the bytes python-docx and unstructured *copy* out of
3550#: ``document.xml`` beyond reading it once: the text, attributes,
3551#: namespace declarations and nodes that one bounded paragraph or cell
3552#: turns into many copies, each priced at what a copy of it was
3553#: measured to keep.
3554#:
3555#: - A paragraph with K rendered page breaks: unstructured's
3556#: ``iter_paragraph_items`` recurses once per break, and each level
3557#: deep-copies the rest of the paragraph twice (python-docx's
3558#: preceding and following fragments) and keeps both until the
3559#: paragraph is done. Runs and hyperlinks leave the fragment they do
3560#: not belong to, so their content is kept up to K + 1 times (80
3561#: breaks before 3 MB of text, a 40 KB file, kept ~230 MB more than
3562#: the same paragraph without them, and 20 breaks before 200,000
3563#: nested empty elements ~530 MB, ~130 bytes a node); everything else
3564#: stays in all 2K copies: the ``w:p`` element with its attributes
3565#: and namespace declarations, its ``w:pPr``, the comments and
3566#: processing instructions in it, the text after it (lxml copies an
3567#: element's tail), and a declaration of each namespace it uses that
3568#: an ancestor declares (libxml2 re-declares those on every copy).
3569#: Each break is therefore counted as two copies of the paragraph,
3570#: of its tail, and of each namespace declaration of the
3571#: ``w:document`` and ``w:body`` elements whose URI an element or
3572#: attribute name in the paragraph uses: 8 breaks before a 2 MB
3573#: namespace URI on the ``w:p``, a 39 KB file, kept ~29 MB, and such
3574#: declarations were not counted before (a URI that long is now
3575#: refused outright, see ``MAX_DOCX_NAMESPACE_URI_CHARS``).
3576#: - A table cell spanning S grid columns: its text is extracted and
3577#: kept S times (counted as S - 1 extra copies, multiplied through
3578#: nested tables), each copy priced at
3579#: ``DOCX_TABLE_COPY_BYTES_PER_TEXT_BYTE`` per byte of its text, since
3580#: unstructured keeps each copy as a string and twice more HTML-escaped
3581#: (in its row's and in the table's HTML): 20 extra copies of a 1 MB
3582#: cell of ``"`` and one emoji, a 38 KB file, kept ~1 GB.
3583#: - A cell continuing a vertical merge: python-docx yields the cell
3584#: its merge starts from, so that cell's text is kept once more per
3585#: continuing row, priced as a spanned cell's copies are. The
3586#: streaming pass cannot resolve which cell that is, so it counts the
3587#: largest cell (times its span) of the earlier rows of the same
3588#: table: 300 continuing rows under a 1 MB cell, a 38 KB file, kept
3589#: ~930 MB and took 21 s, and 20 under a 1 MB cell of ``"`` and one
3590#: emoji kept ~1 GB. A table whose continuing rows times its largest
3591#: earlier cell exceed about 2 MiB of text (``MAX_DOCX_COPIED_BYTES``
3592#: divided by ``DOCX_TABLE_COPY_BYTES_PER_TEXT_BYTE``; 200 continuing
3593#: rows beside a 10 KB cell) is therefore refused, even when the cells
3594#: actually merged are small. A cell continues exactly
3595#: when python-docx and unstructured say so (``tc.vMerge ==
3596#: "continue"``): the first ``w:vMerge`` of the cell's first
3597#: ``w:tcPr`` has ``w:val="continue"`` or no ``w:val``. Any other
3598#: cell, whatever other ``w:vMerge`` or ``w:tcPr`` elements it
3599#: carries, is counted as an ordinary cell, once per spanned column.
3600#:
3601#: Real documents copy a few MB: Word writes about one rendered page
3602#: break per page, in paragraphs of a few KB, and spans and merges of
3603#: small cells. Element, attribute and processing-instruction names
3604#: are not counted: lxml's copies share them through the parser's name
3605#: dictionary (measured: no growth). The count is not a measurement:
3606#: each item is priced at about the most its copy was measured to keep
3607#: (an element ~130 bytes, an attribute ~275, a declaration ~110,
3608#: a byte of lxml text about 1; a byte of table-cell text ~52 at worst,
3609#: ~12 for plain ASCII), and the shapes measured kept less than they
3610#: were counted (20 page breaks before 200,000 nested empty elements
3611#: ~530 MB, counted ~1 GB; 20 before 200 elements of 256 empty
3612#: attributes ~310 MB, counted ~590 MB; 20 extra copies of a 1 MB cell of ``"`` and
3613#: one emoji ~1 GB, counted ~1.3 GB), so a document at this ceiling
3614#: keeps on the order of 128 MiB of copies, not more.
3615MAX_DOCX_COPIED_BYTES = 128 * 1024 * 1024
3617#: Bytes a copied element, comment, processing instruction or
3618#: namespace declaration counts in ``MAX_DOCX_COPIED_BYTES`` besides
3619#: its text, attribute values, prefix and URI: an lxml node (~130 bytes
3620#: per copied empty element, ~110 per copied declaration measured).
3621DOCX_COPIED_BYTES_PER_NODE = 128
3623#: Bytes an attribute counts in ``MAX_DOCX_COPIED_BYTES`` besides its
3624#: value: libxml2 copies it as an attribute node and a text node
3625#: (~275 bytes per empty attribute per copy measured, the same with or
3626#: without a namespace).
3627DOCX_COPIED_BYTES_PER_ATTRIBUTE = 288
3629#: Bytes a byte of table-cell text counts in ``MAX_DOCX_COPIED_BYTES``
3630#: per extra copy of the cell (spanned column or continued merged
3631#: row). unstructured's ``_convert_table_to_html`` keeps every copy as
3632#: a string, then HTML-escaped in its row's string and again in the
3633#: table's, all at once: a ``"`` (one byte of XML text) becomes
3634#: ``"`` (six characters) and one astral character in a string
3635#: makes it four bytes per character, so a byte costs up to 4 + 2 x 24
3636#: = 52 bytes (measured ~49 per extra copy; ~12 for plain ASCII text).
3637DOCX_TABLE_COPY_BYTES_PER_TEXT_BYTE = 64
3639#: Children of python-docx's default styles part, which it creates when
3640#: the document part has no styles relationship; the style lookup
3641#: (``MAX_DOCX_STYLE_WORK``) scans at least that many entries.
3642DOCX_DEFAULT_STYLES_ENTRIES = 6
3644_RELS_NS = "{http://schemas.openxmlformats.org/package/2006/relationships}"
3645_R_ID = (
3646 "{http://schemas.openxmlformats.org/officeDocument/2006/relationships}id"
3647)
3648_RT_OFFICE_DOCUMENT = (
3649 "http://schemas.openxmlformats.org/officeDocument/2006/relationships/"
3650 "officeDocument"
3651)
3652_P_NS = "{http://schemas.openxmlformats.org/presentationml/2006/main}"
3653_W_NS = "{http://schemas.openxmlformats.org/wordprocessingml/2006/main}"
3654_W_HDRFTR_REFS = (f"{_W_NS}headerReference", f"{_W_NS}footerReference")
3657def _skeleton_xml(archive: zipfile.ZipFile, name: str):
3658 """Parse member *name* (held to ``MAX_OPC_SKELETON_PART_BYTES``,
3659 and read through ``_read_counted_xml``: UTF-8, start tags of at
3660 most ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` attributes, no DTD) without
3661 entity resolution or network access; ``None`` if absent."""
3662 from lxml import etree
3664 try:
3665 info = archive.getinfo(name)
3666 except KeyError:
3667 return None
3668 if info.file_size > MAX_OPC_SKELETON_PART_BYTES:
3669 raise DecompressionBombError(
3670 "document package part is larger than "
3671 f"{MAX_OPC_SKELETON_PART_BYTES} bytes"
3672 )
3673 parser = etree.XMLParser(resolve_entities=False, no_network=True)
3674 root = etree.fromstring(_read_counted_xml(archive, info), parser=parser)
3675 _refuse_dtd(root)
3676 return root
3679def _part_rels(
3680 archive: zipfile.ZipFile,
3681 packuri_cls,
3682 partname: str,
3683 *,
3684 skip_missing_targets: bool,
3685) -> dict[str, tuple[str, Optional[str]]]:
3686 """``{rId: (reltype, target partname or None if external)}`` for
3687 *partname*, built the way the loading library builds it: later
3688 duplicates of an rId win, and (python-pptx only,
3689 *skip_missing_targets*) internal relationships whose target is not
3690 in the archive are dropped before that. A relationships part with
3691 more than ``MAX_PART_RELATIONSHIPS`` children is refused."""
3692 members = set(archive.namelist())
3693 root = _skeleton_xml(archive, packuri_cls(partname).rels_uri.membername)
3694 rels: dict[str, tuple[str, Optional[str]]] = {}
3695 if root is None:
3696 return rels
3697 if len(root) > MAX_PART_RELATIONSHIPS: 3697 ↛ 3698line 3697 didn't jump to line 3698 because the condition on line 3697 was never true
3698 raise DecompressionBombError(
3699 f"package relationships part has {len(root)} entries "
3700 f"(ceiling {MAX_PART_RELATIONSHIPS})"
3701 )
3702 base_uri = packuri_cls(partname).baseURI
3703 for rel in root.findall(f"{_RELS_NS}Relationship"):
3704 if rel.get("TargetMode", "Internal") == "External":
3705 target = None
3706 else:
3707 target = packuri_cls.from_rel_ref(base_uri, rel.get("Target"))
3708 if skip_missing_targets and target[1:] not in members:
3709 continue
3710 rels[rel.get("Id")] = (rel.get("Type"), target)
3711 return rels
3714def _check_package_relationships(
3715 archive: zipfile.ZipFile, *, price_walk: bool
3716) -> None:
3717 """Refuse a ``.docx``/``.pptx`` package any of whose relationships
3718 parts (``*.rels`` members, whichever part they belong to, reached
3719 or not) is over ``MAX_OPC_SKELETON_PART_BYTES``, declares a DTD or
3720 has more than ``MAX_PART_RELATIONSHIPS`` entries, whose
3721 relationships parts have more than ``MAX_PACKAGE_RELATIONSHIPS``
3722 entries together, or (*price_walk*, python-docx) whose internal
3723 relationships times its members cost python-docx's relationship
3724 walks more than ``MAX_DOCX_RELATIONSHIP_WALK``. Each part is
3725 streamed and each entry dropped once read, so at most the root and
3726 one entry are held: every element child of the root counts as an
3727 entry, internal unless its ``TargetMode`` is ``External``, as the
3728 libraries read it, and every comment and processing instruction
3729 counts as an entry too. A part with an element below an entry, or
3730 an element with more than ``MAX_RELS_ELEMENT_ATTRIBUTES``
3731 attributes, is refused when that element starts: the OPC schema
3732 gives ``Relationship`` four attributes and text content only, and
3733 without this one entry holding millions of children (16 MiB, about
3734 16 KB deflated, in a part no library reads) was built in full,
3735 costing the guard 3.1 s and ~550 MB per part. Each part is read
3736 through ``_CountedXmlStream``, so lxml never receives a start tag of
3737 more than ``MAX_XML_TAG_ATTRIBUTE_SIGNS`` attributes: ``iterparse``
3738 builds one in full before its start event (1,000,000 attributes
3739 cost this check about 200 MB before)."""
3740 from lxml import etree
3742 total = internal = 0
3743 for info in archive.infolist():
3744 if not info.filename.lower().endswith(".rels"):
3745 continue
3746 if info.file_size > MAX_OPC_SKELETON_PART_BYTES:
3747 raise DecompressionBombError(
3748 "document package part is larger than "
3749 f"{MAX_OPC_SKELETON_PART_BYTES} bytes"
3750 )
3751 entries = depth = 0
3752 with archive.open(info) as stream:
3753 for event, element in etree.iterparse(
3754 _CountedXmlStream(stream),
3755 events=("start", "end", "comment", "pi"),
3756 resolve_entities=False,
3757 no_network=True,
3758 ):
3759 if event == "end":
3760 depth -= 1
3761 if depth == 1:
3762 # Keep no tree: drop each entry once read.
3763 element.clear()
3764 parent = element.getparent()
3765 while element.getprevious() is not None:
3766 del parent[0]
3767 continue
3768 if event == "start":
3769 depth += 1
3770 if depth > 2:
3771 raise DecompressionBombError(
3772 "package relationships entry has child elements"
3773 )
3774 if len(element.attrib) > MAX_RELS_ELEMENT_ATTRIBUTES:
3775 raise DecompressionBombError(
3776 "package relationships element has more than "
3777 f"{MAX_RELS_ELEMENT_ATTRIBUTES} attributes"
3778 )
3779 if depth == 1:
3780 _refuse_dtd(element)
3781 continue
3782 if element.get("TargetMode") != "External":
3783 internal += 1
3784 # An entry, or a comment or processing instruction
3785 # anywhere in the part (each stays in the tree until
3786 # the entry it is in, or the next one, ends).
3787 entries += 1
3788 if entries > MAX_PART_RELATIONSHIPS:
3789 raise DecompressionBombError(
3790 "package relationships part has more than "
3791 f"{MAX_PART_RELATIONSHIPS} entries"
3792 )
3793 total += entries
3794 if total > MAX_PACKAGE_RELATIONSHIPS:
3795 raise DecompressionBombError(
3796 "package relationships parts have more than "
3797 f"{MAX_PACKAGE_RELATIONSHIPS} entries together"
3798 )
3799 if not price_walk or not internal:
3800 return
3801 names = archive.namelist()
3802 steps = (
3803 len(names)
3804 + sum(len(name) * _str_width(name) for name in names)
3805 // DOCX_RELATIONSHIP_WALK_NAME_BYTES_PER_STEP
3806 )
3807 if internal * steps > MAX_DOCX_RELATIONSHIP_WALK:
3808 raise DecompressionBombError(
3809 f"package's {internal} internal relationships over {len(names)} "
3810 f"members cost {internal * steps} steps in python-docx's "
3811 f"relationship walk (ceiling {MAX_DOCX_RELATIONSHIP_WALK})"
3812 )
3815def _str_width(text: str) -> int:
3816 """Bytes per character CPython stores *text* with (PEP 393): 1 when
3817 every code point is below 256, 2 below 65,536, else 4. Comparing
3818 two equal strings of equal length reads ``len * width`` bytes."""
3819 if text.isascii():
3820 return 1
3821 widest = max(text)
3822 if widest <= "\xff":
3823 return 1
3824 return 2 if widest <= "\uffff" else 4
3827def _main_part(
3828 archive: zipfile.ZipFile, packuri_cls, *, skip_missing_targets: bool
3829) -> Optional[str]:
3830 """The package's main document partname, resolved as the library
3831 resolves it, or ``None`` when the library cannot load the package
3832 at all: python-pptx and python-docx both require exactly one
3833 internal officeDocument relationship (``part_with_reltype`` raises
3834 otherwise) whose target is in the archive. Such a package is
3835 refused: the library cannot extract text from it, and a package
3836 without its main part is what a content sniffer may take for
3837 another type (see ``_check_package_type``)."""
3838 rels = _part_rels(
3839 archive, packuri_cls, "/", skip_missing_targets=skip_missing_targets
3840 )
3841 mains = [
3842 target
3843 for reltype, target in rels.values()
3844 if reltype == _RT_OFFICE_DOCUMENT
3845 ]
3846 if len(mains) != 1 or mains[0] is None:
3847 return None
3848 if mains[0][1:] not in set(archive.namelist()): 3848 ↛ 3849line 3848 didn't jump to line 3849 because the condition on line 3848 was never true
3849 return None
3850 return mains[0]
3853_CT_NS = "{http://schemas.openxmlformats.org/package/2006/content-types}"
3854_CT_PREFIX = "application/vnd.openxmlformats-officedocument."
3855_CT_PML_SLIDE = frozenset({_CT_PREFIX + "presentationml.slide+xml"})
3856_CT_PML_SLIDE_LAYOUT = frozenset(
3857 {_CT_PREFIX + "presentationml.slideLayout+xml"}
3858)
3859_CT_PML_SLIDE_MASTER = frozenset(
3860 {_CT_PREFIX + "presentationml.slideMaster+xml"}
3861)
3862#: python-pptx's ``Presentation()`` accepts these main parts only.
3863_CT_PML_MAINS = frozenset(
3864 {
3865 _CT_PREFIX + "presentationml.presentation.main+xml",
3866 "application/vnd.ms-powerpoint.presentation.macroEnabled.main+xml",
3867 }
3868)
3869#: python-docx's ``Document()`` accepts this main part only.
3870_CT_WML_MAINS = frozenset({_CT_PREFIX + "wordprocessingml.document.main+xml"})
3871_CT_WML_STYLES = frozenset({_CT_PREFIX + "wordprocessingml.styles+xml"})
3872_CT_WML_SETTINGS = frozenset({_CT_PREFIX + "wordprocessingml.settings+xml"})
3873_CT_WML_HEADER = frozenset({_CT_PREFIX + "wordprocessingml.header+xml"})
3874_CT_WML_FOOTER = frozenset({_CT_PREFIX + "wordprocessingml.footer+xml"})
3877class _ContentTypes(NamedTuple):
3878 """``[Content_Types].xml`` as python-pptx and python-docx read it:
3879 ``Override`` content types by lower-cased part name and ``Default``
3880 ones by lower-cased extension, a later entry replacing an earlier
3881 one with the same key."""
3883 overrides: dict[str, Optional[str]]
3884 defaults: dict[str, Optional[str]]
3887def _content_types(archive: zipfile.ZipFile) -> _ContentTypes:
3888 """Read the package's content types (see ``_ContentTypes``; the
3889 part is held to ``MAX_OPC_SKELETON_PART_BYTES`` and refused with a
3890 DTD). A package without the part, or with an entry missing its
3891 part name or extension, is refused: both libraries fail to open
3892 it."""
3893 root = _skeleton_xml(archive, "[Content_Types].xml")
3894 if root is None:
3895 raise DecompressionBombError("document package has no content types")
3896 overrides: dict[str, Optional[str]] = {}
3897 defaults: dict[str, Optional[str]] = {}
3898 for kind, key, table in (
3899 ("Override", "PartName", overrides),
3900 ("Default", "Extension", defaults),
3901 ):
3902 for entry in root.findall(f"{_CT_NS}{kind}"):
3903 name = entry.get(key)
3904 if name is None: 3904 ↛ 3905line 3904 didn't jump to line 3905 because the condition on line 3904 was never true
3905 raise DecompressionBombError(
3906 "document package has a malformed content type entry"
3907 )
3908 table[name.lower()] = entry.get("ContentType")
3909 return _ContentTypes(overrides, defaults)
3912def _content_type(types: _ContentTypes, partname: str) -> Optional[str]:
3913 """The content type the libraries give *partname* (an absolute
3914 part name): its ``Override``, else the ``Default`` of its extension
3915 (``PackURI.ext``), else ``None``."""
3916 key = partname.lower()
3917 if key in types.overrides:
3918 return types.overrides[key]
3919 ext = posixpath.splitext(partname)[1]
3920 ext = ext[1:] if ext.startswith(".") else ext
3921 return types.defaults.get(ext.lower())
3924def _require_content_type(
3925 types: _ContentTypes, partname: str, allowed: frozenset[str], what: str
3926) -> None:
3927 """Refuse the package unless *partname*'s content type is in
3928 *allowed*.
3930 python-pptx and python-docx pick a part's class from its content
3931 type, not from the relationship that reaches it, and several
3932 classes answer the same property by following their own
3933 relationship of the same type: a slide whose layout relationship
3934 targets another slide reaches that slide's layout (``SlidePart``
3935 has ``slide_layout`` too), and a document part whose styles or
3936 settings relationship targets another document part reaches that
3937 part's styles or settings. The guard reads the part a relationship
3938 targets as the kind the relationship names, so such a chain would
3939 let python-pptx or python-docx read parts the guard never priced
3940 (a slide over a chained 300-placeholder layout was counted at 1,200
3941 units, the same layout reached directly at about 90,000). A part
3942 the library would build as another class either fails there or is
3943 such a chain, so nothing a document legitimately holds is lost."""
3944 content_type = _content_type(types, partname)
3945 if content_type not in allowed:
3946 raise DecompressionBombError(
3947 f"document package's {what} part has the wrong content type"
3948 )
3951#: Ceiling on python-pptx's placeholder inheritance as unstructured
3952#: drives it: the sum over slides of the slide's placeholders times the
3953#: cost of one inherited read (``_pptx_inherited_read_units``).
3954#: unstructured reads ``top`` and ``left`` of every shape on a slide up
3955#: to three times each (to sort the shapes, and twice more to skip
3956#: off-slide ones); a slide placeholder with no position of its own
3957#: inherits it, and each such read rebuilds the slide part's
3958#: relationships by type, finds the slide layout's placeholder of the
3959#: same ``idx`` by building a proxy for each placeholder of the layout's
3960#: shape tree in turn (two to three XPath evaluations each, every one
3961#: walking the shape's children, its first child's and that child's
3962#: ``p:nvPr``'s), and, when that one has no position either, does the
3963#: same over the slide master's shape tree. Nothing is cached, so a
3964#: slide of P placeholders over a layout of L shapes costs about
3965#: 6 * P * L proxy builds: before the change, 100 placeholders over a
3966#: 100-placeholder layout, a 30 KB file, took 5.2 s and 200 over 200,
3967#: 18.6 s, quadrupling per doubling. A unit measured 400-430 us end to
3968#: end in ``partition_pptx`` (the 200-over-200 slide is 46,000 units,
3969#: 0.51 of the ceiling), so this fan-out at the ceiling is about 40 s.
3970#: Office's default theme, as python-pptx and pandoc ship it, has 5-10
3971#: shapes per layout and 7 on the master, and its title-and-content
3972#: slide two placeholders without a position: 34 units per slide, so
3973#: about 2,600 such slides fit; templates with tens of shapes per
3974#: layout and master fit proportionally fewer (see "Known gaps").
3975MAX_PPTX_PLACEHOLDER_WORK = 90_000
3977#: Nodes below a layout's or master's shapes (``_PptxShapeTree.nodes``)
3978#: that count as one unit of ``MAX_PPTX_PLACEHOLDER_WORK``: each of the
3979#: six reads walks them at ~50 ns per node (a unit is ~420 us), so one
3980#: counted twice for the text before it is priced at about ten times
3981#: that.
3982PPTX_PLACEHOLDER_NODES_PER_UNIT = 256
3984#: Relationships of the slide and layout parts that count as one unit of
3985#: ``MAX_PPTX_PLACEHOLDER_WORK``: python-pptx rebuilds the part's
3986#: relationships by type on every read (100 placeholders on a slide of
3987#: 10,000 relationships took 3.7 s more, ~0.6 us per relationship and
3988#: read), so one is priced at about 3.5 times that.
3989PPTX_PLACEHOLDER_RELATIONSHIPS_PER_UNIT = 32
3991_RT_SLIDE_LAYOUT = (
3992 "http://schemas.openxmlformats.org/officeDocument/2006/relationships/"
3993 "slideLayout"
3994)
3995_RT_SLIDE_MASTER = (
3996 "http://schemas.openxmlformats.org/officeDocument/2006/relationships/"
3997 "slideMaster"
3998)
3999# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4000_P_CSLD = f"{_P_NS}cSld"
4001# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4002_P_SPTREE = f"{_P_NS}spTree"
4003# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4004_P_NVPR = f"{_P_NS}nvPr"
4005# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4006_P_PH = f"{_P_NS}ph"
4009class _PptxShapeTree(NamedTuple):
4010 """What ``_pptx_shape_tree`` counts in one slide-like part."""
4012 #: Children of ``p:cSld/p:spTree`` (shapes and anything else but
4013 #: text).
4014 shapes: int
4015 #: Their children, grandchildren and great-grandchildren (elements,
4016 #: comments and processing instructions; python-pptx's
4017 #: ``./*[1]/p:nvPr/p:ph`` walks some of them), each counted twice
4018 #: for the text node that may precede it.
4019 nodes: int
4020 #: Children holding a ``p:ph`` in a ``p:nvPr`` of one of their own
4021 #: children: python-pptx's placeholders (it looks in the first
4022 #: child only, so this can only overcount).
4023 placeholders: int
4026def _check_pptx_slide_fan_out(content: bytes) -> None:
4027 """Refuse a presentation listing the same slide part more than once,
4028 or whose placeholder inheritance costs more than
4029 ``MAX_PPTX_PLACEHOLDER_WORK``.
4031 python-pptx yields one slide per ``<p:sldId>`` (first ``p:sldIdLst``
4032 of the presentation part), resolved through that part's
4033 relationships, so N entries resolving to one slide part make
4034 unstructured extract that slide N times. Genuine presentations list
4035 each slide part once. A ``<p:sldId>`` without a resolvable internal
4036 relationship is refused too: python-pptx fails on it, so nothing
4037 indexable is lost, and the check never has to guess. Each slide, and
4038 each slide layout a slide with placeholders relates to and each
4039 slide master such a layout relates to, is then streamed once (see
4040 ``_pptx_shape_tree``). Every relationships part of the package is
4041 held to ``MAX_PART_RELATIONSHIPS`` entries, and all of them together
4042 to ``MAX_PACKAGE_RELATIONSHIPS`` (``_check_package_relationships``).
4043 """
4044 try:
4045 import pptx # noqa: F401
4046 except ImportError: # no python-pptx, no .pptx loader
4047 return
4048 try:
4049 # Inside the fail-closed block, as for .xlsx: if an upgrade
4050 # moves the helper, uploads are refused, not left unchecked.
4051 from pptx.opc.packuri import PackURI
4053 with zipfile.ZipFile(io.BytesIO(content)) as archive:
4054 _check_package_relationships(archive, price_walk=False)
4055 types = _content_types(archive)
4056 targets = _pptx_slide_targets(archive, PackURI, types)
4057 repeated = len(targets) != len(set(targets))
4058 work = (
4059 0
4060 if repeated
4061 else _pptx_placeholder_work(
4062 archive, PackURI, targets, MAX_PPTX_PLACEHOLDER_WORK, types
4063 )
4064 )
4065 except DecompressionBombError:
4066 raise
4067 except Exception as exc: # lxml/zipfile failure modes are open-ended
4068 raise DecompressionBombError("malformed presentation package") from exc
4069 if repeated:
4070 raise DecompressionBombError(
4071 "presentation lists the same slide part more than once"
4072 )
4073 if work > MAX_PPTX_PLACEHOLDER_WORK:
4074 raise DecompressionBombError(
4075 "presentation's placeholders cost more than "
4076 f"{MAX_PPTX_PLACEHOLDER_WORK} units of python-pptx's layout and "
4077 "master lookups"
4078 )
4081def _pptx_placeholder_work(
4082 archive: zipfile.ZipFile,
4083 packuri_cls,
4084 slides: list[str],
4085 limit: int,
4086 content_types: Optional[_ContentTypes] = None,
4087) -> int:
4088 """The sum over *slides* of each slide's placeholders times the
4089 units of one inherited read (see ``MAX_PPTX_PLACEHOLDER_WORK``),
4090 returned as soon as it exceeds *limit* (so that no more parts are
4091 streamed than needed to refuse). Every placeholder is counted as
4092 inheriting (whether or not it has a position of its own), every
4093 slide layout a slide relates to (python-pptx fails on more than
4094 one) and every master such a layout relates to are counted at the
4095 costliest, and each part is streamed at most once. Each such layout
4096 and master must have the content type python-pptx builds it from
4097 (see ``_require_content_type``)."""
4098 if content_types is None:
4099 content_types = _content_types(archive)
4100 types = content_types
4101 members = set(archive.namelist())
4102 trees: dict[str, _PptxShapeTree] = {}
4103 rels: dict[str, dict[str, tuple[str, Optional[str]]]] = {}
4105 def tree(partname: str) -> _PptxShapeTree:
4106 if partname not in trees:
4107 trees[partname] = _pptx_shape_tree(archive, partname[1:])
4108 return trees[partname]
4110 def related(
4111 partname: str, reltype: str, allowed: frozenset[str], what: str
4112 ) -> list[str]:
4113 if partname not in rels:
4114 rels[partname] = _part_rels(
4115 archive, packuri_cls, partname, skip_missing_targets=True
4116 )
4117 targets = [
4118 target
4119 for kind, target in rels[partname].values()
4120 if kind == reltype and target is not None and target[1:] in members
4121 ]
4122 for target in targets:
4123 _require_content_type(types, target, allowed, what)
4124 return targets
4126 work = 0
4127 for slide in slides:
4128 placeholders = tree(slide).placeholders
4129 if not placeholders:
4130 continue
4131 read = 0
4132 for layout in related(
4133 slide, _RT_SLIDE_LAYOUT, _CT_PML_SLIDE_LAYOUT, "slide layout"
4134 ):
4135 masters = related(
4136 layout, _RT_SLIDE_MASTER, _CT_PML_SLIDE_MASTER, "slide master"
4137 )
4138 relationships = len(rels[slide]) + len(rels[layout])
4139 candidates: list[Optional[str]] = list(masters) or [None]
4140 for master in candidates:
4141 read = max(
4142 read,
4143 _pptx_inherited_read_units(
4144 tree(layout),
4145 _PptxShapeTree(0, 0, 0)
4146 if master is None
4147 else tree(master),
4148 relationships,
4149 ),
4150 )
4151 work += placeholders * read
4152 if work > limit:
4153 break
4154 return work
4157def _pptx_inherited_read_units(
4158 layout: _PptxShapeTree, master: _PptxShapeTree, relationships: int
4159) -> int:
4160 """Units of ``MAX_PPTX_PLACEHOLDER_WORK`` for one slide placeholder:
4161 the shapes of its layout and master, their nodes per
4162 ``PPTX_PLACEHOLDER_NODES_PER_UNIT`` and the slide's and layout's
4163 *relationships* per ``PPTX_PLACEHOLDER_RELATIONSHIPS_PER_UNIT``
4164 (each rounded up)."""
4165 return (
4166 layout.shapes
4167 + master.shapes
4168 - (-(layout.nodes + master.nodes) // PPTX_PLACEHOLDER_NODES_PER_UNIT)
4169 - (-relationships // PPTX_PLACEHOLDER_RELATIONSHIPS_PER_UNIT)
4170 )
4173def _pptx_shape_tree(archive: zipfile.ZipFile, name: str) -> _PptxShapeTree:
4174 """Count member *name*'s shape tree (see ``_PptxShapeTree``) in one
4175 streaming pass, with no tree kept. A DTD, a namespace URI or
4176 prefix over the caps ``document.xml`` is held to, or more than
4177 ``MAX_XML_OUTSIDE_ROOT_NODES`` comments and processing instructions
4178 outside the root, is refused. An
4179 element's name is read only down to the depth of ``p:ph``, so the
4180 pass is linear in the part's size."""
4181 from lxml import etree
4183 shapes = nodes = placeholders = outside = 0
4184 # Names of the open elements down to p:ph's depth (7), and the depth.
4185 path: list[str] = []
4186 depth = 0
4187 in_tree = has_ph = False
4188 with archive.open(name) as stream:
4189 for event, element in etree.iterparse(
4190 _CountedXmlStream(stream),
4191 events=("start-ns", "start", "end", "comment", "pi"),
4192 resolve_entities=False,
4193 no_network=True,
4194 ):
4195 if event == "start-ns":
4196 prefix, uri = element
4197 if ( 4197 ↛ 4201line 4197 didn't jump to line 4201 because the condition on line 4197 was never true
4198 len(uri) > MAX_DOCX_NAMESPACE_URI_CHARS
4199 or len(prefix) > MAX_DOCX_NAMESPACE_PREFIX_CHARS
4200 ):
4201 raise DecompressionBombError(
4202 "presentation part declares a namespace URI of more "
4203 f"than {MAX_DOCX_NAMESPACE_URI_CHARS} or a prefix of "
4204 f"more than {MAX_DOCX_NAMESPACE_PREFIX_CHARS} "
4205 "characters"
4206 )
4207 continue
4208 if event == "end":
4209 if depth <= 7:
4210 path.pop()
4211 if depth == 3:
4212 in_tree = False
4213 depth -= 1
4214 element.clear(keep_tail=False)
4215 elif event == "start":
4216 depth += 1
4217 if depth == 1:
4218 _refuse_dtd(element)
4219 if depth <= 7:
4220 tag = element.tag
4221 path.append(tag)
4222 if depth == 3:
4223 in_tree = path[1] == _P_CSLD and tag == _P_SPTREE
4224 elif in_tree and depth == 4:
4225 shapes += 1
4226 has_ph = False
4227 elif in_tree:
4228 nodes += 2
4229 if (
4230 depth == 7
4231 and tag == _P_PH
4232 and path[5] == _P_NVPR
4233 and not has_ph
4234 ):
4235 has_ph = True
4236 placeholders += 1
4237 elif depth == 0: 4237 ↛ 4239line 4237 didn't jump to line 4239 because the condition on line 4237 was always true
4238 outside = _note_outside_root_node(outside)
4239 elif in_tree and depth == 3:
4240 # A comment or processing instruction among the shapes,
4241 # which python-pptx iterates over too.
4242 shapes += 1
4243 elif in_tree and 5 <= depth + 1 <= 7:
4244 # A comment or processing instruction among the nodes.
4245 nodes += 2
4246 if event != "start":
4247 # Drop what has been read (comments and processing
4248 # instructions get no end event).
4249 parent = element.getparent()
4250 if parent is not None:
4251 while element.getprevious() is not None:
4252 del parent[0]
4253 return _PptxShapeTree(shapes, nodes, placeholders)
4256def _pptx_slide_targets(
4257 archive: zipfile.ZipFile,
4258 packuri_cls,
4259 content_types: Optional[_ContentTypes] = None,
4260) -> list[str]:
4261 """The slide partname of every ``<p:sldId>``, in order. The main
4262 part and each slide must have the content type python-pptx builds
4263 them from (see ``_require_content_type``)."""
4264 main = _main_part(archive, packuri_cls, skip_missing_targets=True)
4265 if main is None:
4266 raise DecompressionBombError(
4267 "presentation package has no main document part"
4268 )
4269 if content_types is None:
4270 content_types = _content_types(archive)
4271 _require_content_type(content_types, main, _CT_PML_MAINS, "main")
4272 root = _skeleton_xml(archive, main[1:])
4273 if root is None: # pragma: no cover - _main_part checked membership
4274 return []
4275 rels = _part_rels(archive, packuri_cls, main, skip_missing_targets=True)
4276 sld_id_lst = root.find(f"{_P_NS}sldIdLst")
4277 if sld_id_lst is None:
4278 return []
4279 targets = []
4280 for sld_id in sld_id_lst.findall(f"{_P_NS}sldId"):
4281 resolved = rels.get(sld_id.get(_R_ID))
4282 if resolved is None or resolved[1] is None:
4283 raise DecompressionBombError(
4284 "presentation lists a slide it cannot resolve"
4285 )
4286 _require_content_type(
4287 content_types, resolved[1], _CT_PML_SLIDE, "slide"
4288 )
4289 targets.append(resolved[1])
4290 return targets
4293def _check_docx_section_fan_out(content: bytes) -> None:
4294 """Bound python-docx's and unstructured's per-section,
4295 per-paragraph, per-page-break and per-spanned-column work and
4296 header/footer repeats (the known fan-outs; not the document's total
4297 parse cost, see "Known gaps" in the module docstring).
4299 Refused: more than ``MAX_DOCX_SECTIONS`` ``w:sectPr`` elements or
4300 ``MAX_DOCX_PAGE_BREAKS`` ``w:lastRenderedPageBreak`` elements
4301 anywhere in ``document.xml``; a relationships part with
4302 more than ``MAX_PART_RELATIONSHIPS`` entries, relationships parts
4303 with more than ``MAX_PACKAGE_RELATIONSHIPS`` together, or internal
4304 relationships times members over ``MAX_DOCX_RELATIONSHIP_WALK``
4305 (``_check_package_relationships``); a settings or styles
4306 part over ``MAX_OPC_SKELETON_PART_BYTES``, with a DTD, or with more
4307 than ``MAX_DOCX_SETTINGS_CHILDREN`` or ``MAX_DOCX_STYLES`` children
4308 of its root; a ``w:gridSpan``, ``w:gridBefore`` or ``w:gridAfter``
4309 over ``MAX_DOCX_GRID_SPAN`` or that python-docx cannot read; an
4310 element with more than ``MAX_DOCX_ELEMENT_ATTRIBUTES`` attributes
4311 or more than ``MAX_DOCX_NAMESPACES_IN_SCOPE`` namespace declarations
4312 in scope; a body-level paragraph with more than
4313 ``MAX_DOCX_PARAGRAPH_NAMESPACES`` declarations; a declaration whose
4314 URI or prefix is longer than ``MAX_DOCX_NAMESPACE_URI_CHARS`` or
4315 ``MAX_DOCX_NAMESPACE_PREFIX_CHARS``; a table row of more than
4316 ``MAX_DOCX_ROW_CELLS`` cells or ``MAX_DOCX_ROW_GRID_COLUMNS``
4317 columns; the
4318 sum over sections of the body-level blocks up to
4319 each section's end, times the body's child nodes, over
4320 ``MAX_DOCX_SECTION_BLOCK_WORK``; the section count times the linear
4321 per-section work (the body's child nodes, plus the nodes the
4322 ``_sectPrs`` XPath visits divided by
4323 ``DOCX_SECTION_XPATH_NODES_PER_UNIT``, plus the node pairs its union
4324 compares divided by ``DOCX_SECTION_UNION_PAIRS_PER_UNIT``) over
4325 ``MAX_DOCX_SECTION_WORK``; the summed size of the parts the
4326 sections' ``headerReference``/``footerReference`` children point at
4327 (counted once per reference, with the union steps of each read)
4328 over ``MAX_DOCX_HEADER_FOOTER_BYTES``;
4329 the style lookups over ``MAX_DOCX_STYLE_WORK``; the page-break
4330 splitting over ``MAX_DOCX_PAGE_BREAK_WORK``; the text extraction
4331 from runs and table cells (once per spanned column) over
4332 ``MAX_DOCX_RUN_WORK`` (with each cell's paragraphs and tables and
4333 the node-set unions of cells, paragraphs and runs, see
4334 ``DOCX_UNION_STEPS_PER_RUN_WORK_UNIT``, and the merges and sort of
4335 unstructured's page-break union, see
4336 ``DOCX_PAGE_BREAK_UNION_PAIRS_PER_UNIT`` and
4337 ``DOCX_PAGE_BREAK_SORT_STEPS_PER_UNIT``); the bytes copied per page
4338 break, spanned column and continued merged cell over
4339 ``MAX_DOCX_COPIED_BYTES``;
4340 and, since each ceiling is the work one kind alone may cost, any
4341 document whose seven works' shares of their ceilings sum to more
4342 than 1.
4343 Sections are the ``w:sectPr`` elements python-docx enumerates
4344 (``w:body/w:p/w:pPr/w:sectPr`` and ``w:body/w:sectPr``). A
4345 reference that does not resolve to a part in the archive is
4346 refused, as python-docx fails on it. ``document.xml`` is streamed
4347 with no tree kept, so this costs one parse pass over it.
4349 The section, relationship, settings and styles caps bound the
4350 per-section and per-paragraph work python-docx and unstructured do
4351 that is not counted (see ``MAX_DOCX_SECTIONS``); it is bounded by
4352 those caps, not modelled.
4353 """
4354 try:
4355 import docx # noqa: F401
4356 except ImportError: # no python-docx, no .docx loader
4357 return
4358 try:
4359 from docx.opc.packuri import PackURI
4361 with zipfile.ZipFile(io.BytesIO(content)) as archive:
4362 _check_package_relationships(archive, price_walk=True)
4363 stats = _docx_section_stats(archive, PackURI)
4364 except DecompressionBombError:
4365 raise
4366 except Exception as exc: # lxml/zipfile failure modes are open-ended
4367 raise DecompressionBombError(
4368 "malformed word-processing package"
4369 ) from exc
4371 if stats.block_work > MAX_DOCX_SECTION_BLOCK_WORK:
4372 raise DecompressionBombError(
4373 f"document's {stats.sections} sections cost {stats.block_work} "
4374 "block-by-node steps in python-docx's section lookup (ceiling "
4375 f"{MAX_DOCX_SECTION_BLOCK_WORK})"
4376 )
4377 sections = stats.sections
4378 work = (
4379 sections * stats.body_nodes
4380 + sections * stats.xpath_nodes // DOCX_SECTION_XPATH_NODES_PER_UNIT
4381 + sections * stats.union_pairs // DOCX_SECTION_UNION_PAIRS_PER_UNIT
4382 )
4383 if work > MAX_DOCX_SECTION_WORK:
4384 raise DecompressionBombError(
4385 f"document has {sections} sections over {stats.body_nodes} "
4386 f"body nodes, {stats.xpath_nodes} section-lookup nodes and "
4387 f"{stats.union_pairs} section-lookup pairs (work {work}, ceiling "
4388 f"{MAX_DOCX_SECTION_WORK})"
4389 )
4390 if stats.referenced_bytes > MAX_DOCX_HEADER_FOOTER_BYTES:
4391 raise DecompressionBombError(
4392 "document's header/footer references total over "
4393 f"{MAX_DOCX_HEADER_FOOTER_BYTES} bytes"
4394 )
4395 style_work = stats.styled_paragraphs * (
4396 stats.style_entries
4397 + stats.style_attributes // DOCX_STYLE_ATTRIBUTES_PER_ENTRY
4398 + stats.relationships // DOCX_RELATIONSHIPS_PER_STYLE_ENTRY
4399 )
4400 if style_work > MAX_DOCX_STYLE_WORK:
4401 raise DecompressionBombError(
4402 f"document's {stats.styled_paragraphs} paragraphs cost "
4403 f"{style_work} steps in python-docx's style lookup over "
4404 f"{stats.style_entries} styles part entries and "
4405 f"{stats.relationships} relationships (ceiling "
4406 f"{MAX_DOCX_STYLE_WORK})"
4407 )
4408 if stats.page_break_work > MAX_DOCX_PAGE_BREAK_WORK:
4409 raise DecompressionBombError(
4410 "document's rendered page breaks cost "
4411 f"{stats.page_break_work} steps in python-docx's paragraph "
4412 f"splitting (ceiling {MAX_DOCX_PAGE_BREAK_WORK})"
4413 )
4414 if stats.run_work > MAX_DOCX_RUN_WORK:
4415 raise DecompressionBombError(
4416 f"document's runs and table cells cost {stats.run_work} units "
4417 f"of python-docx text extraction (ceiling {MAX_DOCX_RUN_WORK})"
4418 )
4419 if stats.copied_bytes > MAX_DOCX_COPIED_BYTES:
4420 raise DecompressionBombError(
4421 f"document's page breaks, spanned and merged cells copy "
4422 f"{stats.copied_bytes} bytes in python-docx and unstructured "
4423 f"(ceiling {MAX_DOCX_COPIED_BYTES})"
4424 )
4425 # Each ceiling above is the work one kind alone may cost; together
4426 # they share one budget, so a document near several ceilings at
4427 # once costs no more than one at a single ceiling.
4428 shares = (
4429 stats.block_work / MAX_DOCX_SECTION_BLOCK_WORK,
4430 work / MAX_DOCX_SECTION_WORK,
4431 stats.referenced_bytes / MAX_DOCX_HEADER_FOOTER_BYTES,
4432 style_work / MAX_DOCX_STYLE_WORK,
4433 stats.page_break_work / MAX_DOCX_PAGE_BREAK_WORK,
4434 stats.run_work / MAX_DOCX_RUN_WORK,
4435 stats.copied_bytes / MAX_DOCX_COPIED_BYTES,
4436 )
4437 if sum(shares) > 1:
4438 raise DecompressionBombError(
4439 "document's python-docx work takes "
4440 f"{sum(shares):.2f} of the combined budget (section lookup, "
4441 "section work, header/footer bytes, style lookup, page-break "
4442 "splitting, text extraction and copied bytes, each as a share "
4443 "of its ceiling)"
4444 )
4447# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4448_W_BODY = f"{_W_NS}body"
4449# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4450_W_P = f"{_W_NS}p"
4451# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4452_W_PPR = f"{_W_NS}pPr"
4453# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4454_W_SECTPR = f"{_W_NS}sectPr"
4455# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4456_W_TBL = f"{_W_NS}tbl"
4457_W_BLOCKS = (_W_P, _W_TBL)
4458# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4459_W_R = f"{_W_NS}r"
4460_W_RUN_CONTAINERS = (_W_R, f"{_W_NS}hyperlink")
4461# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4462_W_TR = f"{_W_NS}tr"
4463# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4464_W_TC = f"{_W_NS}tc"
4465# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4466_W_TRPR = f"{_W_NS}trPr"
4467# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4468_W_TCPR = f"{_W_NS}tcPr"
4469# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4470_W_GRID_SPAN = f"{_W_NS}gridSpan"
4471# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4472_W_GRID_BEFORE = f"{_W_NS}gridBefore"
4473_W_GRID_VALUES = (_W_GRID_SPAN, _W_GRID_BEFORE, f"{_W_NS}gridAfter")
4474# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4475_W_VMERGE = f"{_W_NS}vMerge"
4476# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4477_W_VAL = f"{_W_NS}val"
4478# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4479_W_LAST_RENDERED_PAGE_BREAK = f"{_W_NS}lastRenderedPageBreak"
4480# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4481_W_HYPERLINK = f"{_W_NS}hyperlink"
4482# nosemgrep: semgrep.rules.sql-string-concatenation, reason: OOXML qualified name constant; no SQL sink
4483_W_DRAWING = f"{_W_NS}drawing"
4484_WP_INLINE = (
4485 "{http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing}"
4486 "inline"
4487)
4488#: The paths of unstructured's page-break union branches below the
4489#: root, by depth of the ``w:lastRenderedPageBreak`` (see
4490#: ``MAX_DOCX_PAGE_BREAKS``).
4491_W_PAGE_BREAK_BRANCHES = {
4492 5: (_W_BODY, _W_P, _W_R),
4493 6: (_W_BODY, _W_P, _W_HYPERLINK, _W_R),
4494 8: (_W_BODY, _W_TBL, _W_TR, _W_TC, _W_P, _W_R),
4495 9: (_W_BODY, _W_TBL, _W_TR, _W_TC, _W_P, _W_HYPERLINK, _W_R),
4496}
4497#: The branches of ``CT_R.text``'s union, one bit each.
4498_W_RUN_TEXT_BRANCHES = {
4499 f"{_W_NS}{name}": 1 << bit
4500 for bit, name in enumerate(
4501 ("br", "cr", "noBreakHyphen", "ptab", "t", "tab")
4502 )
4503}
4504#: Elements whose children python-docx selects with a union.
4505_W_UNION_PARENTS = frozenset({_W_TC, _W_P, _W_R})
4506_RT_SETTINGS = (
4507 "http://schemas.openxmlformats.org/officeDocument/2006/relationships/"
4508 "settings"
4509)
4510_RT_STYLES = (
4511 "http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles"
4512)
4515class _DocxSectionStats(NamedTuple):
4516 """What ``_docx_section_stats`` counts in one streaming pass."""
4518 #: ``w:sectPr`` elements anywhere in ``document.xml``.
4519 sect_prs: int
4520 #: The sections python-docx enumerates (N1 + N2 below).
4521 sections: int
4522 #: Child nodes of the body: elements, comments, processing
4523 #: instructions and (non-empty) text.
4524 body_nodes: int
4525 #: Node visits of one ``_sectPrs`` XPath evaluation.
4526 xpath_nodes: int
4527 #: N1 * N2: paragraph-level times body-level section ``w:sectPr``.
4528 union_pairs: int
4529 #: Sum over sections of the body-level ``w:p``/``w:tbl`` up to the
4530 #: section's end, times ``body_nodes``.
4531 block_work: int
4532 #: Header/footer bytes, counted once per reference, plus the union
4533 #: steps of each read (``DOCX_UNION_STEPS_PER_HEADER_BYTE``).
4534 referenced_bytes: int
4535 #: Body-level paragraphs with a ``w:r`` or ``w:hyperlink`` child
4536 #: (the only ones whose text can be non-empty), plus the
4537 #: ``w:lastRenderedPageBreak`` elements inside them: an upper bound
4538 #: on the paragraph fragments whose style unstructured looks up.
4539 styled_paragraphs: int
4540 #: Sum over body-level paragraphs of K * (K + 3) * S**2, K being the
4541 #: paragraph's ``w:lastRenderedPageBreak`` elements and S the largest
4542 #: child-node count of the paragraph or of any element in it.
4543 page_break_work: int
4544 #: Sum over body-level paragraphs and tables of (K + 1) *
4545 #: (``DOCX_RUN_WORK_ITEM_WEIGHT`` * (``w:r``, ``w:tr`` and ``w:tc``
4546 #: elements in it) + child nodes of its ``w:r`` elements) + K * (its
4547 #: descendant nodes) // ``DOCX_FRAGMENT_NODES_PER_RUN_WORK_UNIT``
4548 #: (K = 0 for a table). Within a table, a cell's work (with one
4549 #: unit per ``DOCX_CELL_NODES_PER_UNIT`` of its descendant nodes) is
4550 #: counted once per grid column it spans (through nested tables), a
4551 #: cell continuing a vertical merge instead counts the largest cell
4552 #: of the earlier rows, and a row counts its
4553 #: ``gridBefore``/``gridAfter`` columns
4554 #: (``DOCX_GRID_PADDING_PER_RUN_WORK_UNIT`` per unit) and
4555 #: python-docx's steps up the merges its cells continue (see
4556 #: ``_docx_merge_steps``). A cell also counts its ``w:p`` and
4557 #: ``w:tbl`` children (``DOCX_CELL_BLOCK_ITEM_WEIGHT``), and every
4558 #: cell, paragraph and run the steps of its union
4559 #: (``DOCX_UNION_STEPS_PER_RUN_WORK_UNIT``; a body-level paragraph's
4560 #: with the runs in its runs' inline drawings as a third branch,
4561 #: and the nodes in those drawings), multiplied with the rest of its
4562 #: work; without sections, the body's union is added once, and the
4563 #: pairs merged and the steps sorted by unstructured's page-break
4564 #: union are added once (``DOCX_PAGE_BREAK_UNION_PAIRS_PER_UNIT``,
4565 #: ``DOCX_PAGE_BREAK_SORT_STEPS_PER_UNIT``).
4566 run_work: int
4567 #: Bytes copied beyond one read (see ``MAX_DOCX_COPIED_BYTES``):
4568 #: per body-level paragraph, 2K times its bytes, its tail and the
4569 #: ancestors' namespace declarations of the URIs it uses; per
4570 #: spanned cell, (span - 1) times its text bytes; per continuing
4571 #: merged cell, the text bytes of the largest cell of the earlier
4572 #: rows of its table; cell text times
4573 #: ``DOCX_TABLE_COPY_BYTES_PER_TEXT_BYTE``.
4574 copied_bytes: int
4575 #: Children of the styles part's root, at least
4576 #: ``DOCX_DEFAULT_STYLES_ENTRIES`` (python-docx's default part).
4577 style_entries: int
4578 #: Attributes of those children.
4579 style_attributes: int
4580 #: Relationships of the document part.
4581 relationships: int
4584def _docx_section_stats(archive: zipfile.ZipFile, packuri_cls):
4585 """Section statistics of the main document part (see
4586 ``_DocxSectionStats``), from one streaming pass, after checking the
4587 relationships, settings and styles parts.
4589 The union pairs are N1 * N2, N1 being the ``w:body/w:p/w:pPr/w:sectPr``
4590 elements and N2 the ``w:body/w:sectPr`` elements: libxml2 compares
4591 every node of one union branch's result with every node of the
4592 other's to drop duplicates.
4594 The block work is, summed over sections, the body-level
4595 ``w:p``/``w:tbl`` elements from the start of the body through the
4596 paragraph holding a paragraph-level ``w:sectPr`` (or before a
4597 body-level one), which is what python-docx's ``preceding-sibling``
4598 lookup returns for that section, times the body's child nodes,
4599 which bound the sibling walk of each comparison sorting them (see
4600 ``MAX_DOCX_SECTION_BLOCK_WORK``). Blocks and nodes are counted
4601 across all ``w:body`` elements, which can only overcount. Text
4602 nodes are counted from the text before each body child and after
4603 the last one, so ``element.clear`` keeps tails until then.
4605 The node visits are those one evaluation of python-docx's
4606 ``/w:document/w:body/w:p/w:pPr/w:sectPr | /w:document/w:body/w:sectPr``
4607 makes: every element, comment and processing instruction that is a
4608 child of the document node, of the root element, or of a ``w:body``
4609 child of it (each counted twice, since both union branches walk
4610 these levels), or of a body-level ``w:p`` or of that paragraph's
4611 ``w:pPr`` (counted once). Text nodes are not counted; each sits
4612 beside a counted node or alone in its parent. Entity reference
4613 nodes, for which ``iterparse`` emits no event, cannot occur: a
4614 ``document.xml`` with a DTD is refused in its bytes, before lxml
4615 reads it (``_CountedXmlStream``).
4616 Counting does not check the root's tag, which only adds nodes the
4617 XPath would not visit.
4619 Per body-level paragraph and table (python-docx's block items; the
4620 content of ``w:sdt`` and other body-level elements is not read), the
4621 child nodes of every element in it are counted as python-docx's tree
4622 has them, text included: elements, comments and processing
4623 instructions, the text before each and the text after the last.
4624 From those come the page-break and run work (see
4625 ``MAX_DOCX_PAGE_BREAK_WORK`` and ``MAX_DOCX_RUN_WORK``) and the
4626 paragraphs whose style is looked up (``MAX_DOCX_STYLE_WORK``).
4627 ``w:lastRenderedPageBreak`` elements are counted at any depth of a
4628 paragraph and ``w:r`` elements at any depth of a block, though
4629 python-docx reads only some of them, which can only overcount. The
4630 bytes of each block (``MAX_DOCX_COPIED_BYTES``) are its text, its
4631 attribute values, its namespace declarations (prefix and URI, from
4632 the ``start-ns`` events), ``DOCX_COPIED_BYTES_PER_NODE`` per node
4633 and per declaration and ``DOCX_COPIED_BYTES_PER_ATTRIBUTE`` per
4634 attribute; the extra copies of a table cell count its text bytes
4635 times ``DOCX_TABLE_COPY_BYTES_PER_TEXT_BYTE``; a paragraph's copies also count the text after
4636 it and the declarations on the ``w:document`` and ``w:body``
4637 elements of the namespace URIs its element and attribute names
4638 use. A ``w:tc`` anywhere in a block weighs its content by its
4639 ``w:gridSpan`` (the largest under any of its ``w:tcPr``, where
4640 python-docx reads the first ``w:gridSpan`` of the first
4641 ``w:tcPr``: never less), unless it continues a vertical merge as
4642 python-docx reads it (the first ``w:vMerge`` of its first
4643 ``w:tcPr``, with ``w:val`` absent or ``continue``): such a cell
4644 adds the largest earlier cell of its table instead, which the cell
4645 python-docx actually yields cannot exceed, and is not itself a
4646 candidate for that largest cell. A row python-docx reads (a direct
4647 ``w:tr`` child of a table) also counts python-docx's steps up the
4648 vertical merges its cells continue (``_docx_merge_steps``), from the
4649 spans and ``gridBefore`` as python-docx reads them.
4651 Streaming stops with a refusal as soon as ``MAX_DOCX_SECTIONS`` or
4652 ``MAX_DOCX_PAGE_BREAKS`` is exceeded, at the first grid value over ``MAX_DOCX_GRID_SPAN``, at
4653 the first element with more than ``MAX_DOCX_ELEMENT_ATTRIBUTES``
4654 attributes (counted before any is read), at the first with more
4655 than ``MAX_DOCX_NAMESPACES_IN_SCOPE`` namespace declarations in
4656 scope, at the first namespace declaration whose URI or prefix is
4657 over its cap (before any name under it is read), at the first
4658 paragraph over ``MAX_DOCX_PARAGRAPH_NAMESPACES``, at the first row
4659 cell over ``MAX_DOCX_ROW_CELLS``, at the first other row child over
4660 ``MAX_DOCX_ROW_OTHER_CHILDREN``, at the first row after more than
4661 ``MAX_DOCX_TABLE_ROW_GAP`` other table children and at the end of
4662 the first row over ``MAX_DOCX_ROW_GRID_COLUMNS``.
4664 The pass reads each element's name (lxml builds it, namespace URI
4665 included, on every read) at most twice and each attribute name at
4666 most once, and each text, attribute value and comment at most
4667 twice, so its time is linear in the size of ``document.xml``, a
4668 name costing up to ``MAX_DOCX_NAMESPACE_URI_CHARS`` more per read
4669 than its own length; the state it keeps is bounded by the depth (at
4670 most 256, libxml2's limit), the namespace declarations in scope
4671 (``pending_ns`` is refused as soon as it would exceed them) and the
4672 open tables' last rows."""
4673 from lxml import etree
4675 main = _main_part(archive, packuri_cls, skip_missing_targets=False)
4676 if main is None:
4677 raise DecompressionBombError(
4678 "word-processing package has no main document part"
4679 )
4680 types = _content_types(archive)
4681 _require_content_type(types, main, _CT_WML_MAINS, "main")
4682 rels = _part_rels(archive, packuri_cls, main, skip_missing_targets=False)
4683 _docx_skeleton_entries(
4684 archive,
4685 rels,
4686 _RT_SETTINGS,
4687 MAX_DOCX_SETTINGS_CHILDREN,
4688 "settings",
4689 types,
4690 _CT_WML_SETTINGS,
4691 )
4692 styles, style_attributes = _docx_skeleton_entries(
4693 archive,
4694 rels,
4695 _RT_STYLES,
4696 MAX_DOCX_STYLES,
4697 "styles",
4698 types,
4699 _CT_WML_STYLES,
4700 )
4701 sizes = {info.filename: info.file_size for info in archive.infolist()}
4702 body_nodes = xpath_nodes = referenced_bytes = outside = 0
4703 # Blocks seen so far, and their sum over sections.
4704 sect_prs = blocks = section_blocks = 0
4705 # w:sectPr in body-level paragraph properties, and body-level ones.
4706 paragraph_sections = body_sections = 0
4707 styled_paragraphs = page_break_work = run_work = copied_bytes = 0
4708 path: list[str] = []
4709 # Child nodes seen so far of each element in ``path``.
4710 children: list[int] = []
4711 # The body-level paragraph or table being read, if any (0 if none,
4712 # else 3, its depth): whether it is a paragraph with a w:r or
4713 # w:hyperlink child, its page breaks, descendant nodes, largest
4714 # child-node count, bytes (nodes, attribute values and text, see
4715 # ``MAX_DOCX_COPIED_BYTES``) and text bytes as read once.
4716 block_depth = 0
4717 is_paragraph = has_run = False
4718 page_breaks = block_nodes = widest = block_bytes = block_text = 0
4719 # Run-work units and text bytes of the block as unstructured reads
4720 # ``row.cells`` (python-docx shares a cell's objects between its
4721 # yields, so only its text is copied):
4722 # frames[0] for the block, one more per w:tc open in it, which is
4723 # folded into its parent's frame, times its span, when it ends.
4724 frames: list[list[int]] = []
4725 # The w:tc, w:tbl and w:tr elements open in the block.
4726 cells: list[_OpenCell] = []
4727 tables: list[_OpenTable] = []
4728 rows: list[_OpenRow] = []
4729 # Namespace declarations in the body-level paragraph being read.
4730 paragraph_ns = 0
4731 # Union branch counts of each element in ``path`` whose children
4732 # python-docx selects with a union (see
4733 # ``DOCX_UNION_STEPS_PER_RUN_WORK_UNIT``), None for the others: a
4734 # cell's w:p and w:tbl, a paragraph's w:r and w:hyperlink, a run's
4735 # text branches (as a bit set) and their count.
4736 unions: list[Optional[list[int]]] = []
4737 # The body's w:p and w:tbl children (its union is read once when the
4738 # document has no sections).
4739 body_paragraphs = body_tables = 0
4740 # Header/footer references per part.
4741 references: dict[str, int] = {}
4742 # w:lastRenderedPageBreak elements anywhere, and those in each branch
4743 # of unstructured's page-break union, by depth (see
4744 # MAX_DOCX_PAGE_BREAKS).
4745 all_breaks = 0
4746 break_branches = dict.fromkeys(_W_PAGE_BREAK_BRANCHES, 0)
4747 # The union's breaks below each element in ``path``, and the sum
4748 # over elements of those breaks (weighted for the sort's binary
4749 # insertion) times the element's child nodes (see
4750 # DOCX_PAGE_BREAK_SORT_STEPS_PER_UNIT).
4751 breaks_below: list[int] = []
4752 break_sort_steps = 0
4753 # The w:drawing elements, and the wp:inline elements below one, open
4754 # in a w:r child of the body-level paragraph being read: the runs
4755 # under such an inline are the third branch of unstructured's
4756 # paragraph union (see DOCX_UNION_STEPS_PER_RUN_WORK_UNIT).
4757 drawings = inlines = 0
4759 def visits(depth: int) -> int:
4760 """Section-XPath visits of a node at *depth* (1 = a child of the
4761 document node), given the ancestors in ``path[:depth - 1]``."""
4762 if depth <= 2:
4763 return 2
4764 if path[1] != _W_BODY: 4764 ↛ 4765line 4764 didn't jump to line 4765 because the condition on line 4764 was never true
4765 return 0
4766 if depth == 3:
4767 return 2
4768 if path[2] != _W_P:
4769 return 0
4770 if depth == 4:
4771 return 1
4772 if depth == 5 and path[3] == _W_PPR:
4773 return 1
4774 return 0
4776 def text_before(node) -> Optional[str]:
4777 """The text node that precedes *node* among its siblings, if
4778 any: python-docx keeps text that is not whitespace, and some
4779 that is."""
4780 previous = node.getprevious()
4781 text: Optional[str] = (
4782 node.getparent().text if previous is None else previous.tail
4783 )
4784 return text
4786 # The namespace declarations announced by ``start-ns`` events for
4787 # the next element, as (URI, bytes), and the bytes of those on the
4788 # w:document and w:body elements by URI: libxml2 re-declares one on
4789 # each copy of a paragraph that uses its URI.
4790 pending_ns: list[tuple[str, int]] = []
4791 ancestor_ns: dict[str, int] = {}
4792 # The namespace URIs of ``ancestor_ns`` that element and attribute
4793 # names in the body-level paragraph being read use (at most one
4794 # entry per declaration there, so this stays small however many
4795 # names use them).
4796 used_uris: set[str] = set()
4797 # Copies kept of the text after the last body-level paragraph,
4798 # added when that text is seen.
4799 tail_copies = 0
4801 attribute_values = etree.XPath("@*", smart_strings=False)
4803 def node_bytes(node, is_element: bool) -> int:
4804 """What copying *node* (an element, comment or processing
4805 instruction) costs, without its children or the text around
4806 them: a node, and per attribute its node and its value."""
4807 size = DOCX_COPIED_BYTES_PER_NODE
4808 if is_element:
4809 # At most MAX_DOCX_ELEMENT_ATTRIBUTES, checked at its start.
4810 count = len(node.attrib)
4811 if count:
4812 size += count * DOCX_COPIED_BYTES_PER_ATTRIBUTE
4813 values = (
4814 node.attrib.values()
4815 if count <= _DOCX_ATTRIBUTE_VALUES_SCAN
4816 else attribute_values(node)
4817 )
4818 for value in values:
4819 size += _text_bytes(value)
4820 else:
4821 size += _text_bytes(node.text)
4822 return size
4824 # Depth of the section-defining w:sectPr being read, if any:
4825 # python-docx reads header/footer references from its direct
4826 # children only.
4827 section_depth = 0
4828 # Namespace declarations in scope at each element of ``path``.
4829 in_scope: list[int] = []
4830 with archive.open(main[1:]) as stream:
4831 for event, element in etree.iterparse(
4832 _CountedXmlStream(stream),
4833 events=("start-ns", "start", "end", "comment", "pi"),
4834 resolve_entities=False,
4835 no_network=True,
4836 ):
4837 if event == "start-ns":
4838 prefix, uri = element
4839 # Before any name under it is read (see
4840 # MAX_DOCX_NAMESPACE_URI_CHARS).
4841 if (
4842 len(uri) > MAX_DOCX_NAMESPACE_URI_CHARS
4843 or len(prefix) > MAX_DOCX_NAMESPACE_PREFIX_CHARS
4844 ):
4845 raise DecompressionBombError(
4846 "document declares a namespace URI of more than "
4847 f"{MAX_DOCX_NAMESPACE_URI_CHARS} or a prefix of more "
4848 f"than {MAX_DOCX_NAMESPACE_PREFIX_CHARS} characters"
4849 )
4850 pending_ns.append(
4851 (
4852 uri,
4853 DOCX_COPIED_BYTES_PER_NODE
4854 + _text_bytes(prefix)
4855 + _text_bytes(uri),
4856 )
4857 )
4858 # The element they belong to is checked at its start;
4859 # refusing here already keeps pending_ns as small.
4860 scope = in_scope[-1] if in_scope else 0
4861 if scope + len(pending_ns) > MAX_DOCX_NAMESPACES_IN_SCOPE:
4862 raise DecompressionBombError(
4863 "document has more than "
4864 f"{MAX_DOCX_NAMESPACES_IN_SCOPE} namespace declarations "
4865 "in scope"
4866 )
4867 continue
4868 if event in ("comment", "pi"):
4869 if not path:
4870 # Outside the root, where python-docx never looks.
4871 outside = _note_outside_root_node(outside)
4872 continue
4873 xpath_nodes += visits(len(path) + 1)
4874 before = text_before(element)
4875 if children: 4875 ↛ 4877line 4875 didn't jump to line 4877 because the condition on line 4875 was always true
4876 children[-1] += 1 + bool(before)
4877 if len(path) == 2 and path[1] == _W_BODY:
4878 body_nodes += 1 + bool(before)
4879 if tail_copies:
4880 copied_bytes += tail_copies * _text_bytes(before)
4881 tail_copies = 0
4882 if block_depth:
4883 text = _text_bytes(before)
4884 frames[-1][1] += text
4885 block_text += text
4886 block_bytes += text + node_bytes(element, False)
4887 _note_docx_table_child(len(path) + 1, None, tables, rows)
4888 # Comments and PIs get no end event, so drop the
4889 # siblings before them here too.
4890 parent = element.getparent()
4891 if parent is not None: 4891 ↛ 4894line 4891 didn't jump to line 4894 because the condition on line 4891 was always true
4892 while element.getprevious() is not None:
4893 del parent[0]
4894 continue
4895 # Each element's name is read once per event: lxml builds it
4896 # afresh, URI included, on every read.
4897 tag = element.tag
4898 if event == "end":
4899 if len(path) == section_depth:
4900 section_depth = 0
4901 # Inside an inline drawing of one of the paragraph's runs
4902 # (the inline itself included): its child nodes are
4903 # walked when the third branch's runs are sorted.
4904 in_inline = bool(inlines)
4905 if drawings and len(path) > block_depth + 1:
4906 # The same test as at the element's start: the
4907 # drawings open above it have not changed.
4908 if tag == _W_DRAWING:
4909 drawings -= 1
4910 elif tag == _WP_INLINE:
4911 inlines -= 1
4912 # The text after the element's last child (or all of
4913 # its text, if it has none).
4914 last = element[-1] if len(element) else None
4915 trailing = element.text if last is None else last.tail
4916 if len(path) == 2 and tag == _W_BODY:
4917 body_nodes += bool(trailing)
4918 if tail_copies: 4918 ↛ 4919line 4918 didn't jump to line 4919 because the condition on line 4918 was never true
4919 copied_bytes += tail_copies * _text_bytes(trailing)
4920 tail_copies = 0
4921 own = children.pop() + bool(trailing)
4922 counter = unions.pop()
4923 below = breaks_below.pop()
4924 if below:
4925 break_sort_steps += own * (
4926 below
4927 + DOCX_PAGE_BREAK_SORT_INSERTION_WALKS
4928 * min(below, DOCX_PAGE_BREAK_SORT_INSERTION_RUN)
4929 )
4930 if breaks_below:
4931 breaks_below[-1] += below
4932 if in_inline and block_depth:
4933 paragraph_union = unions[block_depth - 1]
4934 if paragraph_union is not None: 4934 ↛ 4936line 4934 didn't jump to line 4936 because the condition on line 4934 was always true
4935 paragraph_union[3] += own
4936 if block_depth:
4937 text = _text_bytes(trailing)
4938 frames[-1][1] += text
4939 block_text += text
4940 block_bytes += text
4941 block_nodes += own
4942 widest = max(widest, own)
4943 extra = 0
4944 if counter is not None:
4945 extra = _union_units(_union_steps(tag, counter, own))
4946 if tag == _W_TC:
4947 extra += DOCX_CELL_BLOCK_ITEM_WEIGHT * (
4948 counter[0] + counter[1]
4949 )
4950 _fold_docx_text_work(
4951 tag,
4952 len(path),
4953 own,
4954 frames,
4955 cells,
4956 tables,
4957 rows,
4958 block_nodes,
4959 extra,
4960 )
4961 if len(path) == block_depth:
4962 block_depth = 0
4963 units, weighted_text = frames.pop()
4964 if has_run:
4965 styled_paragraphs += 1 + page_breaks
4966 page_break_work += (
4967 page_breaks * (page_breaks + 3) * widest * widest
4968 )
4969 run_work += (page_breaks + 1) * units + (
4970 page_breaks
4971 * block_nodes
4972 // DOCX_FRAGMENT_NODES_PER_RUN_WORK_UNIT
4973 )
4974 # Extra copies of table cells (spanned columns,
4975 # continued merges), as unstructured's HTML
4976 # keeps them; two fragments per break, each a
4977 # deep copy of the paragraph (its tail included)
4978 # that re-declares the ancestors' namespaces it
4979 # uses.
4980 copied_bytes += DOCX_TABLE_COPY_BYTES_PER_TEXT_BYTE * (
4981 weighted_text - block_text
4982 ) + (
4983 2
4984 * page_breaks
4985 * (
4986 block_bytes
4987 + sum(ancestor_ns[uri] for uri in used_uris)
4988 )
4989 )
4990 tail_copies = 2 * page_breaks
4991 path.pop()
4992 in_scope.pop()
4993 # Nothing is read after an element's start event but
4994 # the text before the next sibling, so the tree is
4995 # dropped as it is built, tails kept until then.
4996 element.clear(keep_tail=True)
4997 parent = element.getparent()
4998 if parent is not None:
4999 while element.getprevious() is not None:
5000 del parent[0]
5001 continue
5002 # Counted, not read: reading an element's attribute values
5003 # is quadratic in their number.
5004 if len(element.attrib) > MAX_DOCX_ELEMENT_ATTRIBUTES:
5005 raise DecompressionBombError(
5006 "document has an element with more than "
5007 f"{MAX_DOCX_ELEMENT_ATTRIBUTES} attributes"
5008 )
5009 declared = len(pending_ns)
5010 in_scope.append((in_scope[-1] if in_scope else 0) + declared)
5011 if in_scope[-1] > MAX_DOCX_NAMESPACES_IN_SCOPE: 5011 ↛ 5012line 5011 didn't jump to line 5012 because the condition on line 5011 was never true
5012 raise DecompressionBombError(
5013 "document has more than "
5014 f"{MAX_DOCX_NAMESPACES_IN_SCOPE} namespace declarations "
5015 "in scope"
5016 )
5017 before = None
5018 if not path:
5019 _refuse_dtd(element)
5020 else:
5021 before = text_before(element)
5022 children[-1] += 1 + bool(before)
5023 path.append(tag)
5024 children.append(0)
5025 unions.append(None)
5026 breaks_below.append(0)
5027 depth = len(path)
5028 xpath_nodes += visits(depth)
5029 if tag == _W_LAST_RENDERED_PAGE_BREAK:
5030 all_breaks += 1
5031 if all_breaks > MAX_DOCX_PAGE_BREAKS:
5032 raise DecompressionBombError(
5033 "document has more than "
5034 f"{MAX_DOCX_PAGE_BREAKS} rendered page breaks"
5035 )
5036 branch = _W_PAGE_BREAK_BRANCHES.get(depth)
5037 if branch is not None and tuple(path[1:-1]) == branch:
5038 break_branches[depth] += 1
5039 breaks_below[-1] = 1
5040 ns_bytes = 0
5041 if pending_ns:
5042 for uri, size in pending_ns:
5043 ns_bytes += size
5044 if depth <= 2:
5045 ancestor_ns[uri] = ancestor_ns.get(uri, 0) + size
5046 del pending_ns[:]
5047 if tag == _W_SECTPR:
5048 sect_prs += 1
5049 if sect_prs > MAX_DOCX_SECTIONS:
5050 raise DecompressionBombError(
5051 "document has more than "
5052 f"{MAX_DOCX_SECTIONS} sections (w:sectPr elements)"
5053 )
5054 elif tag in _W_GRID_VALUES:
5055 _note_docx_grid_value(element, tag, path, cells, rows)
5056 elif tag == _W_TCPR:
5057 # python-docx reads a cell's first w:tcPr only...
5058 if cells and cells[-1].depth == depth - 1: 5058 ↛ 5084line 5058 didn't jump to line 5084 because the condition on line 5058 was always true
5059 cells[-1].tc_pr = 1 if cells[-1].tc_pr == 0 else 2
5060 elif tag == _W_TRPR:
5061 # ...and a row's first w:trPr.
5062 if rows and rows[-1].depth == depth - 1: 5062 ↛ 5084line 5062 didn't jump to line 5084 because the condition on line 5062 was always true
5063 row = rows[-1]
5064 row.tr_pr = 1 if row.tr_pr == 0 else 2
5065 if row.tr_pr == 1:
5066 row.tr_pr_nodes = -block_nodes
5067 # The children find() visits up to it.
5068 row.tr_pr_index = children[-2]
5069 elif tag == _W_VMERGE:
5070 # The first w:vMerge of the cell's first w:tcPr, whose
5071 # missing w:val means "continue" (``tc.vMerge ==
5072 # "continue"``).
5073 if (
5074 cells
5075 and cells[-1].depth == depth - 2
5076 and path[-2] == _W_TCPR
5077 and cells[-1].tc_pr == 1
5078 and not cells[-1].merge_read
5079 ):
5080 cells[-1].merge_read = True
5081 cells[-1].continues = (
5082 element.get(_W_VAL, "continue") == "continue"
5083 )
5084 if depth == 3 and path[1] == _W_BODY:
5085 body_nodes += 1 + bool(before)
5086 body_paragraphs += tag == _W_P
5087 body_tables += tag == _W_TBL
5088 if tail_copies:
5089 copied_bytes += tail_copies * _text_bytes(before)
5090 tail_copies = 0
5091 if tag in _W_BLOCKS:
5092 blocks += 1
5093 block_depth = depth
5094 is_paragraph = tag == _W_P
5095 has_run = False
5096 page_breaks = block_nodes = widest = 0
5097 drawings = inlines = 0
5098 block_bytes = block_text = paragraph_ns = 0
5099 used_uris.clear()
5100 frames.append([0, 0])
5101 del cells[:], tables[:], rows[:]
5102 # The text before the block is not part of it.
5103 before = None
5104 elif block_depth and is_paragraph:
5105 if depth == 4 and tag in _W_RUN_CONTAINERS:
5106 has_run = True
5107 elif tag == _W_LAST_RENDERED_PAGE_BREAK:
5108 page_breaks += 1
5109 if block_depth:
5110 parent_union = unions[-2]
5111 if parent_union is not None:
5112 _note_union_child(path[-2], tag, parent_union)
5113 if tag in _W_UNION_PARENTS:
5114 unions[-1] = [0, 0, 0, 0]
5115 if (
5116 is_paragraph
5117 and depth > block_depth + 1
5118 and path[block_depth] == _W_R
5119 ):
5120 if tag == _W_DRAWING:
5121 drawings += 1
5122 elif tag == _WP_INLINE:
5123 if drawings:
5124 inlines += 1
5125 elif tag == _W_R and inlines:
5126 # A run of the paragraph union's third branch.
5127 paragraph_union = unions[block_depth - 1]
5128 if paragraph_union is not None: 5128 ↛ 5132line 5128 didn't jump to line 5132 because the condition on line 5128 was always true
5129 paragraph_union[2] += 1
5130 # Before the element is itself recorded as an open
5131 # table, row or cell.
5132 _note_docx_table_child(depth, tag, tables, rows)
5133 if tag == _W_TC:
5134 frames.append([0, 0])
5135 nodes_before = 0
5136 if rows and rows[-1].depth == depth - 1:
5137 row = rows[-1]
5138 row.count += 1
5139 if row.count > MAX_DOCX_ROW_CELLS:
5140 raise DecompressionBombError(
5141 "document has a table row of more than "
5142 f"{MAX_DOCX_ROW_CELLS} cells"
5143 )
5144 # The row's nodes before this cell: its earlier
5145 # children (text included) and their content.
5146 nodes_before = (
5147 block_nodes - row.nodes + children[-2] - 1
5148 )
5149 cells.append(_OpenCell(depth, nodes_before, block_nodes))
5150 elif tag == _W_TBL:
5151 tables.append(_OpenTable(depth))
5152 elif tag == _W_TR:
5153 # Rows python-docx reads: direct children of a table.
5154 table_row = bool(tables) and tables[-1].depth == depth - 1
5155 rows.append(_OpenRow(depth, table_row, block_nodes))
5156 if table_row:
5157 # The table's children before this row and after
5158 # the last one python-docx reads (text, this row
5159 # and, for the first, w:tblPr and w:tblGrid
5160 # included), which _tr_above walks.
5161 table = tables[-1]
5162 rows[-1].gap = children[-2] - table.row_children
5163 table.row_children = children[-2]
5164 text = _text_bytes(before)
5165 frames[-1][1] += text
5166 block_text += text
5167 block_bytes += text + node_bytes(element, True) + ns_bytes
5168 if is_paragraph:
5169 if declared:
5170 paragraph_ns += declared
5171 if paragraph_ns > MAX_DOCX_PARAGRAPH_NAMESPACES:
5172 raise DecompressionBombError(
5173 "document has a paragraph with more than "
5174 f"{MAX_DOCX_PARAGRAPH_NAMESPACES} namespace "
5175 "declarations"
5176 )
5177 if ancestor_ns: 5177 ↛ 5181line 5177 didn't jump to line 5181 because the condition on line 5177 was always true
5178 _note_used_uris(
5179 tag, element.keys(), ancestor_ns, used_uris
5180 )
5181 if tag == _W_SECTPR and (
5182 (depth == 3 and path[1] == _W_BODY)
5183 or (depth == 5 and path[1:4] == [_W_BODY, _W_P, _W_PPR])
5184 ):
5185 if depth == 3:
5186 body_sections += 1
5187 else:
5188 paragraph_sections += 1
5189 # Through the enclosing paragraph (already counted), or
5190 # the blocks before a body-level w:sectPr.
5191 section_blocks += blocks
5192 section_depth = depth
5193 elif (
5194 section_depth
5195 and depth == section_depth + 1
5196 and tag in _W_HDRFTR_REFS
5197 ):
5198 resolved = rels.get(element.get(_R_ID))
5199 if (
5200 resolved is None
5201 or resolved[1] is None
5202 or resolved[1][1:] not in sizes
5203 ):
5204 raise DecompressionBombError(
5205 "document references a header or footer it "
5206 "cannot resolve"
5207 )
5208 if tag == _W_HDRFTR_REFS[0]:
5209 _require_content_type(
5210 types, resolved[1], _CT_WML_HEADER, "header"
5211 )
5212 else:
5213 _require_content_type(
5214 types, resolved[1], _CT_WML_FOOTER, "footer"
5215 )
5216 referenced_bytes += sizes[resolved[1][1:]]
5217 references[resolved[1][1:]] = (
5218 references.get(resolved[1][1:], 0) + 1
5219 )
5220 if (
5221 not paragraph_sections + body_sections
5222 and body_paragraphs
5223 and body_tables
5224 ):
5225 # Without sections, unstructured reads the body's own union.
5226 run_work += _union_units(blocks * body_nodes, reads=1)
5227 # The merges of unstructured's page-break union (see
5228 # MAX_DOCX_PAGE_BREAKS): each branch against the branches before it.
5229 merged = break_pairs = 0
5230 for count in break_branches.values():
5231 break_pairs += merged * count
5232 merged += count
5233 if break_pairs:
5234 run_work += -(-break_pairs // DOCX_PAGE_BREAK_UNION_PAIRS_PER_UNIT)
5235 # Two non-empty branches (else there are no pairs): the merged
5236 # set may be out of document order, and sorting it walks the
5237 # siblings after the breaks.
5238 run_work += -(-break_sort_steps // DOCX_PAGE_BREAK_SORT_STEPS_PER_UNIT)
5239 if referenced_bytes <= MAX_DOCX_HEADER_FOOTER_BYTES:
5240 # Each reference reads its part's unions once (see
5241 # DOCX_UNION_STEPS_PER_HEADER_BYTE); a sum already over the
5242 # ceiling is refused without reading them.
5243 for name, count in references.items():
5244 referenced_bytes = min(
5245 referenced_bytes
5246 + _docx_part_union_steps(archive, name)
5247 * count
5248 // DOCX_UNION_STEPS_PER_HEADER_BYTE,
5249 _SATURATED,
5250 )
5251 return _DocxSectionStats(
5252 sect_prs=sect_prs,
5253 sections=paragraph_sections + body_sections,
5254 body_nodes=body_nodes,
5255 xpath_nodes=xpath_nodes,
5256 union_pairs=paragraph_sections * body_sections,
5257 block_work=section_blocks * body_nodes,
5258 referenced_bytes=referenced_bytes,
5259 styled_paragraphs=styled_paragraphs,
5260 page_break_work=page_break_work,
5261 run_work=run_work,
5262 copied_bytes=copied_bytes,
5263 style_entries=max(styles, DOCX_DEFAULT_STYLES_ENTRIES),
5264 style_attributes=style_attributes,
5265 relationships=len(rels),
5266 )
5269#: Work and byte counts saturate here: nested spans multiply, and a
5270#: count past every ceiling needs no more digits.
5271_SATURATED = 1 << 64
5274def _note_union_child(parent: str, tag: str, counter: list[int]) -> None:
5275 """Count *tag*, a child of *parent* (a cell, a paragraph, a run or a
5276 header/footer root), in *counter*, its parent's union branches (see
5277 ``DOCX_UNION_STEPS_PER_RUN_WORK_UNIT``)."""
5278 if parent == _W_P:
5279 if tag == _W_R:
5280 counter[0] += 1
5281 elif tag == _W_HYPERLINK:
5282 counter[1] += 1
5283 elif parent == _W_R:
5284 bit = _W_RUN_TEXT_BRANCHES.get(tag)
5285 if bit:
5286 counter[0] |= bit
5287 counter[1] += 1
5288 elif tag == _W_P:
5289 counter[0] += 1
5290 elif tag == _W_TBL:
5291 counter[1] += 1
5294def _union_steps(tag: str, counter: list[int], own: int) -> int:
5295 """Steps of one evaluation of the union over the *own* child nodes
5296 of a *tag* element whose branches *counter* counted: per branch
5297 merged after the first non-empty one, the nodes selected times the
5298 child nodes (or times the nodes selected, when a branch selects
5299 nodes that are not children: a body-level paragraph's runs in inline
5300 drawings, whose sort also walks the nodes in those drawings,
5301 ``counter[3]``), and nothing with fewer than two non-empty
5302 branches."""
5303 if tag == _W_R:
5304 branches = bin(counter[0]).count("1")
5305 selected = counter[1]
5306 else:
5307 branches = sum(1 for count in counter[:3] if count)
5308 selected = sum(counter[:3])
5309 if branches < 2:
5310 return 0
5311 steps = (branches - 1) * selected * max(own, selected)
5312 if len(counter) > 3 and counter[2]:
5313 # A body-level paragraph's runs in inline drawings: sorting
5314 # them walks the nodes of those drawings (``counter[3]``).
5315 steps += selected * counter[3]
5316 return steps
5319def _union_units(steps: int, reads: int = DOCX_UNION_READS) -> int:
5320 """Units of ``MAX_DOCX_RUN_WORK`` for *steps* union steps evaluated
5321 *reads* times (rounded up)."""
5322 if not steps:
5323 return 0
5324 return min(
5325 -(-steps * reads // DOCX_UNION_STEPS_PER_RUN_WORK_UNIT), _SATURATED
5326 )
5329def _docx_part_union_steps(archive: zipfile.ZipFile, name: str) -> int:
5330 """Union steps of one read of header/footer part *name*: its root's
5331 ``./w:p | ./w:tbl``, and the union of every cell, paragraph and run
5332 in it (see ``DOCX_UNION_STEPS_PER_RUN_WORK_UNIT``), each evaluated
5333 once per read. Streamed with no tree kept; a DTD, or a namespace
5334 URI or prefix over its cap, is refused as in ``document.xml``, and
5335 so are more than ``MAX_XML_OUTSIDE_ROOT_NODES`` comments and
5336 processing instructions outside the root."""
5337 from lxml import etree
5339 steps = outside = 0
5340 path: list[str] = []
5341 children: list[int] = []
5342 unions: list[Optional[list[int]]] = []
5343 with archive.open(name) as stream:
5344 for event, element in etree.iterparse(
5345 _CountedXmlStream(stream),
5346 events=("start-ns", "start", "end", "comment", "pi"),
5347 resolve_entities=False,
5348 no_network=True,
5349 ):
5350 if event == "start-ns":
5351 prefix, uri = element
5352 if ( 5352 ↛ 5356line 5352 didn't jump to line 5356 because the condition on line 5352 was never true
5353 len(uri) > MAX_DOCX_NAMESPACE_URI_CHARS
5354 or len(prefix) > MAX_DOCX_NAMESPACE_PREFIX_CHARS
5355 ):
5356 raise DecompressionBombError(
5357 "header or footer declares a namespace URI of more "
5358 f"than {MAX_DOCX_NAMESPACE_URI_CHARS} or a prefix of "
5359 f"more than {MAX_DOCX_NAMESPACE_PREFIX_CHARS} "
5360 "characters"
5361 )
5362 continue
5363 if event == "end":
5364 last = element[-1] if len(element) else None
5365 trailing = element.text if last is None else last.tail
5366 own = children.pop() + bool(trailing)
5367 counter = unions.pop()
5368 tag = path.pop()
5369 if counter is not None:
5370 steps += _union_steps(tag, counter, own)
5371 elif not path:
5372 if event == "start":
5373 _refuse_dtd(element)
5374 path.append(element.tag)
5375 children.append(0)
5376 unions.append([0, 0])
5377 else:
5378 # A comment or processing instruction outside the
5379 # root.
5380 outside = _note_outside_root_node(outside)
5381 continue
5382 else:
5383 previous = element.getprevious()
5384 text = (
5385 element.getparent().text
5386 if previous is None
5387 else previous.tail
5388 )
5389 children[-1] += 1 + bool(text)
5390 if event == "start": 5390 ↛ 5400line 5390 didn't jump to line 5400 because the condition on line 5390 was always true
5391 tag = element.tag
5392 if unions[-1] is not None:
5393 _note_union_child(path[-1], tag, unions[-1])
5394 path.append(tag)
5395 children.append(0)
5396 unions.append([0, 0] if tag in _W_UNION_PARENTS else None)
5397 continue
5398 # An ended element, comment or processing instruction: drop
5399 # the siblings before it (tails are kept until then).
5400 if event == "end": 5400 ↛ 5402line 5400 didn't jump to line 5402 because the condition on line 5400 was always true
5401 element.clear(keep_tail=True)
5402 parent = element.getparent()
5403 if parent is not None:
5404 while element.getprevious() is not None:
5405 del parent[0]
5406 return steps
5409def _note_used_uris(
5410 tag: str, attributes: list[str], declared: dict[str, int], used: set[str]
5411) -> None:
5412 """Add to *used* the namespace URIs in *declared* that the element
5413 name *tag* and the *attributes* names (Clark notation) use."""
5414 for name in (tag, *attributes):
5415 if name[:1] == "{":
5416 uri = name[1 : name.index("}")]
5417 if uri in declared:
5418 used.add(uri)
5421class _OpenCell:
5422 """A ``w:tc`` open in the block being read."""
5424 __slots__ = (
5425 "depth",
5426 "span",
5427 "read_span",
5428 "continues",
5429 "tc_pr",
5430 "merge_read",
5431 "span_read",
5432 "nodes_before",
5433 "start_nodes",
5434 )
5436 def __init__(self, depth: int, nodes_before: int, start_nodes: int) -> None:
5437 self.depth = depth
5438 # The largest w:gridSpan under any of its w:tcPr (the weight of
5439 # its work, never less than python-docx's), and the span
5440 # python-docx reads: the first w:gridSpan of its first w:tcPr.
5441 self.span = 1
5442 self.read_span = 1
5443 # Whether it continues a vertical merge as python-docx reads it.
5444 self.continues = False
5445 # 0: no w:tcPr seen yet, 1: in its first, 2: past its first;
5446 # and whether that first one's w:vMerge and w:gridSpan were read.
5447 self.tc_pr = 0
5448 self.merge_read = False
5449 self.span_read = False
5450 # Child and descendant nodes of its row before it.
5451 self.nodes_before = nodes_before
5452 # The block's node count when the cell started.
5453 self.start_nodes = start_nodes
5456class _OpenRow:
5457 """A ``w:tr`` open in the block being read."""
5459 __slots__ = (
5460 "depth",
5461 "padding",
5462 "cells",
5463 "count",
5464 "columns",
5465 "tr_pr",
5466 "grid_before",
5467 "before_read",
5468 "nodes",
5469 "others",
5470 "gap",
5471 "tr_pr_nodes",
5472 "tr_pr_index",
5473 )
5475 def __init__(self, depth: int, table_row: bool, nodes: int) -> None:
5476 self.depth = depth
5477 # Its gridBefore + gridAfter columns (any w:trPr, negative ones
5478 # pad nothing).
5479 self.padding = 0
5480 # For a row python-docx reads (a direct child of a table), per
5481 # direct w:tc: (the span python-docx reads, whether it continues
5482 # a vertical merge, the row's nodes before it).
5483 self.cells: Optional[list[tuple[int, bool, int]]] = (
5484 [] if table_row else None
5485 )
5486 # Direct w:tc children, and the columns they span (largest
5487 # spans, at least one each).
5488 self.count = 0
5489 self.columns = 0
5490 # 0: no w:trPr seen yet, 1: in its first, 2: past its first; and
5491 # python-docx's grid_before (first w:gridBefore of the first
5492 # w:trPr) once read.
5493 self.tr_pr = 0
5494 self.grid_before = 0
5495 self.before_read = False
5496 # The block's node count when the row started.
5497 self.nodes = nodes
5498 # Children other than w:tc elements (elements, comments and
5499 # processing instructions; see MAX_DOCX_ROW_OTHER_CHILDREN).
5500 self.others = 0
5501 # For a row python-docx reads, its table's child nodes from the
5502 # last such row before it (excluded) through it (included).
5503 self.gap = 0
5504 # The nodes in its first w:trPr (where grid_before is looked
5505 # up), once that has ended, and its child nodes up to and
5506 # including that w:trPr (0 if it has none).
5507 self.tr_pr_nodes = 0
5508 self.tr_pr_index = 0
5511class _OpenTable:
5512 """A ``w:tbl`` open in the block being read."""
5514 __slots__ = (
5515 "depth",
5516 "largest_units",
5517 "largest_text",
5518 "above",
5519 "others",
5520 "row_children",
5521 )
5523 def __init__(self, depth: int) -> None:
5524 self.depth = depth
5525 # The largest cell so far that does not continue a merge (its
5526 # run-work units and text bytes, times its span).
5527 self.largest_units = 0
5528 self.largest_text = 0
5529 # The last row python-docx reads, as ``tc_at_grid_offset``
5530 # searches it: {grid offset: (index of the cell found there,
5531 # merge-step units of that cell if it continues a merge, else
5532 # None)}, its cell count and its nodes.
5533 self.above: Optional[
5534 tuple[dict[int, tuple[int, Optional[int]]], int, int]
5535 ] = None
5536 # Children other than w:tr elements since the last w:tr child
5537 # (elements, comments and processing instructions; see
5538 # MAX_DOCX_TABLE_ROW_GAP), and its child nodes (text included)
5539 # through the last w:tr child python-docx reads.
5540 self.others = 0
5541 self.row_children = 0
5544def _note_docx_table_child(
5545 depth: int,
5546 tag: Optional[str],
5547 tables: list[_OpenTable],
5548 rows: list[_OpenRow],
5549) -> None:
5550 """Count a node starting at *depth* (an element named *tag*, or a
5551 comment or processing instruction if None) among the children of
5552 the open table or row it belongs to, refusing a row python-docx
5553 reads with more than ``MAX_DOCX_ROW_OTHER_CHILDREN`` children other
5554 than ``w:tc`` and a ``w:tr`` child of a table after more than
5555 ``MAX_DOCX_TABLE_ROW_GAP`` other children since the last."""
5556 if tables and tables[-1].depth == depth - 1:
5557 table = tables[-1]
5558 if tag != _W_TR:
5559 table.others += 1
5560 elif table.others > MAX_DOCX_TABLE_ROW_GAP:
5561 raise DecompressionBombError(
5562 "document has a table with more than "
5563 f"{MAX_DOCX_TABLE_ROW_GAP} children before or between "
5564 "its rows"
5565 )
5566 else:
5567 table.others = 0
5568 elif rows and rows[-1].depth == depth - 1 and tag != _W_TC:
5569 row = rows[-1]
5570 if row.cells is not None:
5571 row.others += 1
5572 if row.others > MAX_DOCX_ROW_OTHER_CHILDREN:
5573 raise DecompressionBombError(
5574 "document has a table row with more than "
5575 f"{MAX_DOCX_ROW_OTHER_CHILDREN} children that are not "
5576 "cells"
5577 )
5580def _docx_merge_steps(
5581 row: _OpenRow,
5582 row_cells: list[tuple[int, bool, int]],
5583 table: _OpenTable,
5584 nodes: int,
5585 row_children: int,
5586) -> int:
5587 """Run-work units of python-docx's steps up vertical merges for
5588 *row_cells*, the cells of *row* (which ends with the block's node
5589 count at *nodes* and has *row_children* child nodes), and record
5590 *row* on *table* as the row above the next.
5592 A cell continuing a merge at grid offset O (``grid_before`` plus
5593 the spans, as python-docx reads them, of the cells before it) takes
5594 one step to the row above, where ``tc_at_grid_offset`` returns the
5595 first cell starting exactly at O unless an earlier one starts after
5596 O (it stops there); if that cell continues too, its own steps
5597 follow. Each step is priced at the cells and nodes it walks (see
5598 ``DOCX_MERGE_STEP_WEIGHT``): the row's nodes before the cell, its
5599 children up to its first ``w:trPr`` (all of them if it has none)
5600 and the nodes in that ``w:trPr`` (``grid_before``'s two ``find``
5601 calls; nodes before the cell may be counted twice), the
5602 table's children from the row above through this one
5603 (``_tr_above``'s sibling walk) and the row above's nodes.
5604 Without a row above, or a cell at O, python-docx raises after the
5605 step, which is priced the same."""
5606 walked = (row.tr_pr_index or row_children) + row.tr_pr_nodes + row.gap
5607 above = table.above
5608 found: dict[int, tuple[int, Optional[int]]] = {}
5609 highest: Optional[int] = None
5610 offset = row.grid_before
5611 total = 0
5612 for index, (span, continues, before) in enumerate(row_cells):
5613 steps: Optional[int] = None
5614 if continues:
5615 if above is None:
5616 steps = _merge_step(index, before + walked)
5617 else:
5618 cells_above, count_above, nodes_above = above
5619 hit = cells_above.get(offset)
5620 if hit is None:
5621 steps = _merge_step(
5622 index + count_above, before + walked + nodes_above
5623 )
5624 else:
5625 steps = _merge_step(
5626 index + hit[0], before + walked + nodes_above
5627 )
5628 if hit[1] is not None:
5629 steps += hit[1]
5630 steps = min(steps, _SATURATED)
5631 total = min(total + steps, _SATURATED)
5632 if highest is None or offset > highest:
5633 found[offset] = (index, steps)
5634 highest = offset
5635 offset += span
5636 table.above = (found, len(row_cells), nodes - row.nodes)
5637 return total
5640def _merge_step(cells: int, nodes: int) -> int:
5641 """Run-work units of one step up a vertical merge that reads the
5642 spans of *cells* cells and walks *nodes* nodes."""
5643 return (
5644 DOCX_MERGE_STEP_WEIGHT
5645 + cells // DOCX_MERGE_CELLS_PER_UNIT
5646 + nodes // DOCX_MERGE_NODES_PER_UNIT
5647 )
5650def _text_bytes(text: Optional[str]) -> int:
5651 """UTF-8 size of *text* (0 for ``None``), as lxml stores it."""
5652 if not text:
5653 return 0
5654 return len(text) if text.isascii() else len(text.encode("utf-8"))
5657def _note_docx_grid_value(element, tag, path, cells, rows) -> None:
5658 """Refuse a ``w:gridSpan``/``w:gridBefore``/``w:gridAfter`` whose
5659 value python-docx cannot read or that exceeds
5660 ``MAX_DOCX_GRID_SPAN`` (anywhere in ``document.xml``), and record
5661 it on the open cell or row it belongs to: a ``w:gridSpan`` child of
5662 a cell's ``w:tcPr`` (the largest, if several, and the one
5663 python-docx reads), and ``gridBefore`` / ``gridAfter`` children of
5664 a row's ``w:trPr`` (negative ones pad nothing; python-docx's
5665 ``grid_before`` is the first ``gridBefore`` of the first
5666 ``w:trPr``)."""
5667 value = element.get(_W_VAL)
5668 if value is None:
5669 raise DecompressionBombError(
5670 "document has a table grid value python-docx cannot read"
5671 )
5672 if len(value) > _MAX_DOCX_GRID_VALUE_CHARS:
5673 # int() of a long digit string is not linear in its length.
5674 raise DecompressionBombError(
5675 "document has a table grid value of more than "
5676 f"{_MAX_DOCX_GRID_VALUE_CHARS} characters"
5677 )
5678 try:
5679 # python-docx converts the attribute with int() itself.
5680 number = int(value)
5681 except ValueError:
5682 raise DecompressionBombError(
5683 "document has a table grid value python-docx cannot read"
5684 ) from None
5685 if number > MAX_DOCX_GRID_SPAN:
5686 raise DecompressionBombError(
5687 f"document has a table cell spanning or skipping {number} grid "
5688 f"columns (ceiling {MAX_DOCX_GRID_SPAN})"
5689 )
5690 depth = len(path)
5691 if tag == _W_GRID_SPAN:
5692 if cells and cells[-1].depth == depth - 2 and path[-2] == _W_TCPR: 5692 ↛ exitline 5692 didn't return from function '_note_docx_grid_value' because the condition on line 5692 was always true
5693 cell = cells[-1]
5694 cell.span = max(cell.span, number)
5695 if cell.tc_pr == 1 and not cell.span_read:
5696 cell.span_read = True
5697 cell.read_span = number
5698 elif rows and rows[-1].depth == depth - 2 and path[-2] == _W_TRPR: 5698 ↛ exitline 5698 didn't return from function '_note_docx_grid_value' because the condition on line 5698 was always true
5699 row = rows[-1]
5700 row.padding += max(number, 0)
5701 if tag == _W_GRID_BEFORE and row.tr_pr == 1 and not row.before_read:
5702 row.before_read = True
5703 row.grid_before = number
5706def _fold_docx_text_work(
5707 tag, depth, own, frames, cells, tables, rows, nodes, extra=0
5708) -> None:
5709 """Add the element that ends at *depth* (with *own* child nodes;
5710 the block has *nodes* nodes so far) to the run-work units and text
5711 bytes of ``frames``: ``DOCX_RUN_WORK_ITEM_WEIGHT`` units for a run
5712 (plus its children), row or cell (plus one per
5713 ``DOCX_CELL_NODES_PER_UNIT`` of its descendant nodes); a row's
5714 ``gridBefore`` and ``gridAfter`` columns and python-docx's steps up
5715 the vertical merges its cells continue (``_docx_merge_steps``); and
5716 an ending
5717 cell's whole frame folded into its parent's, times its span, or,
5718 for a cell continuing a vertical merge, plus the largest cell of its
5719 table so far (python-docx yields the merge's first cell in its place
5720 when unstructured reads ``row.cells``). A row spanning more than
5721 ``MAX_DOCX_ROW_GRID_COLUMNS`` columns is refused."""
5722 units = text = 0
5723 if tag == _W_TC and cells and cells[-1].depth == depth:
5724 cell = cells.pop()
5725 units, text = frames.pop()
5726 # Its descendant nodes, read once per yield (see
5727 # DOCX_CELL_NODES_PER_UNIT).
5728 units += (
5729 DOCX_RUN_WORK_ITEM_WEIGHT
5730 + (nodes - cell.start_nodes) // DOCX_CELL_NODES_PER_UNIT
5731 + extra
5732 )
5733 extra = 0
5734 if rows and rows[-1].depth == depth - 1:
5735 row = rows[-1]
5736 row.columns += max(cell.span, 1)
5737 if row.cells is not None:
5738 row.cells.append(
5739 (cell.read_span, cell.continues, cell.nodes_before)
5740 )
5741 if cell.continues:
5742 if tables: 5742 ↛ 5777line 5742 didn't jump to line 5777 because the condition on line 5742 was always true
5743 units += tables[-1].largest_units
5744 text += tables[-1].largest_text
5745 else:
5746 span = max(cell.span, 1)
5747 units = min(units * span, _SATURATED)
5748 text = min(text * span, _SATURATED)
5749 if tables: 5749 ↛ 5777line 5749 didn't jump to line 5777 because the condition on line 5749 was always true
5750 table = tables[-1]
5751 table.largest_units = max(table.largest_units, units)
5752 table.largest_text = max(table.largest_text, text)
5753 elif tag == _W_R:
5754 units = DOCX_RUN_WORK_ITEM_WEIGHT + own
5755 elif tag == _W_TR:
5756 units = DOCX_RUN_WORK_ITEM_WEIGHT
5757 if rows and rows[-1].depth == depth: 5757 ↛ 5777line 5757 didn't jump to line 5777 because the condition on line 5757 was always true
5758 row = rows.pop()
5759 if row.padding + row.columns > MAX_DOCX_ROW_GRID_COLUMNS:
5760 raise DecompressionBombError(
5761 "document has a table row spanning more than "
5762 f"{MAX_DOCX_ROW_GRID_COLUMNS} grid columns"
5763 )
5764 units += -(-row.padding // DOCX_GRID_PADDING_PER_RUN_WORK_UNIT)
5765 # A row python-docx reads is a direct child of tables[-1].
5766 if row.cells is not None and tables:
5767 units += _docx_merge_steps(
5768 row, row.cells, tables[-1], nodes, own
5769 )
5770 elif tag == _W_TRPR and rows and rows[-1].depth == depth - 1:
5771 row = rows[-1]
5772 if row.tr_pr == 1:
5773 # Its first w:trPr ends: the nodes in it.
5774 row.tr_pr_nodes += nodes
5775 elif tag == _W_TBL and tables and tables[-1].depth == depth:
5776 tables.pop()
5777 frame = frames[-1]
5778 frame[0] = min(frame[0] + units + extra, _SATURATED)
5779 frame[1] = min(frame[1] + text, _SATURATED)
5782def _docx_skeleton_entries(
5783 archive: zipfile.ZipFile,
5784 rels: dict[str, tuple[str, Optional[str]]],
5785 reltype: str,
5786 ceiling: int,
5787 what: str,
5788 types: _ContentTypes,
5789 allowed: frozenset[str],
5790) -> tuple[int, int]:
5791 """Hold the document part's *reltype* target (the settings or the
5792 styles part) to the skeleton treatment, and return the children of
5793 its root and their attributes ((0, 0) if there is none).
5795 python-docx resolves both parts through the document part's
5796 relationship of that type (and makes a small default one when there
5797 is none). unstructured reads ``settings.odd_and_even_pages_header_footer``
5798 twice per section, each a search of the settings root's children,
5799 and a paragraph's style about four times per paragraph, each a
5800 Python loop over the styles root's ``w:style`` children that reads
5801 two attributes of each (see ``MAX_DOCX_STYLE_WORK``). Every internal
5802 relationship of *reltype* whose target is in the archive is parsed
5803 (``MAX_OPC_SKELETON_PART_BYTES``, DTD refused) and refused if its
5804 root has more than *ceiling* children (elements, comments and
5805 processing instructions). More than one such relationship makes
5806 python-docx fail; checking each, and returning the largest, is
5807 simply conservative. Each must have the content type in *allowed*
5808 (see ``_require_content_type``): python-docx builds a document part
5809 from a styles or settings target typed as one, whose own styles or
5810 settings it then reads instead."""
5811 largest = (0, 0)
5812 for rel_type, target in rels.values():
5813 if rel_type != reltype or target is None:
5814 continue
5815 root = _skeleton_xml(archive, target[1:])
5816 if root is None: 5816 ↛ 5817line 5816 didn't jump to line 5817 because the condition on line 5816 was never true
5817 continue
5818 _require_content_type(types, target, allowed, what)
5819 if len(root) > ceiling:
5820 raise DecompressionBombError(
5821 f"document {what} part has {len(root)} entries "
5822 f"(ceiling {ceiling})"
5823 )
5824 attributes = sum(
5825 len(child.attrib) for child in root if isinstance(child.tag, str)
5826 )
5827 largest = (
5828 max(largest[0], len(root)),
5829 max(largest[1], attributes),
5830 )
5831 return largest