Coverage for src/local_deep_research/utilities/pdf_stream_release.py: 93%

1"""Release what the PDF libraries cache between the pages of a text walk. 

2 

3Both PDF libraries keep, for the life of the document, everything they 

4decode or parse while extracting a page, and closing a pdfplumber page 

5does not release it: 

6 

7- decoded streams. pdfminer (under pdfplumber) stores a stream's 

8 decompressed bytes on the ``PDFStream`` object (``.data``), which its 

9 object cache and the page objects keep reachable; pypdf stores a 

10 decoded copy on each ``EncodedStreamObject`` (``.decoded_self``), 

11 which its reader's object cache keeps reachable. A crafted page whose 

12 content stream is 20 MB of spaces compresses to ~20 KB, yields no 

13 characters and costs ~0.17 s of CPU, so a walk to the page ceiling 

14 held ~10 GB from a ~10 MB file. 

15- parsed objects. Every dictionary and array a page resolves (fonts, 

16 their ``/Widths``, resources) stays in the object cache, and a 

17 pdfminer font object keeps its own copy of the widths. A page whose 

18 font points at a 500,000-entry ``/Widths`` array held ~60 MB more per 

19 page, at ~2 s of CPU per page. 

20- font tables. pdfminer builds every font in a page's resources (and 

21 in a Form XObject's, on every ``Do`` that invokes it), with tables it 

22 expands from short ranges: a ``/ToUnicode`` CMap's ``<0000> <FFFF> 

23 <0000> beginbfrange`` becomes a 65,536-entry dictionary, and every 

24 font builds its own copy even when they all point at one ~100-byte 

25 compressed CMap stream. A font that is an object of its own is kept 

26 in the resource manager's font cache; one written inline in a 

27 resource dictionary is built again on every use and cached nowhere. 

28 Either way the parser that builds each map keeps it, and the decoded 

29 CMap bytes, in a reference cycle, so dropping the font does not free 

30 them, and parsing them creates too few tracked objects for the 

31 automatic collector to run often. Twenty pages of five such fonts held 

32 ~840 MB, from an 18 KB file or, with the fonts inline, a 13 KB one. 

33 Each font also builds its own widths table, copying the /Widths array 

34 even when many fonts share one (six fonts a page sharing one 

35 200,000-entry array grew ~97 MB a page); a Type 1 font without 

36 /Encoding hands a fresh copy of its /FontFile's first /Length1 bytes 

37 to a header parser that keeps it in a reference cycle (three fonts a 

38 page sharing a 20 MB /FontFile of spaces, a ~30 KB file, grew ~57 MB 

39 a page); the tokens a crafted /ToUnicode CMap or font header leaves 

40 on its parser's stack hold up to ~170 bytes per decoded byte, in the 

41 same cycle; and a CID font joins its /CIDSystemInfo /Registry and 

42 /Ordering into a new string of its own (five fonts a page sharing a 

43 20 MB /Registry string, a 25 KB file, grew ~95 MB a page). 

44 

45A walker creates one releaser per document and calls ``before_page()`` 

46just before each page's ``extract_text()`` and ``after_page()`` once the 

47page is done. A lazy pdfplumber walk (``utilities/pdf_page_walk``) calls 

48``before_page()`` before it builds each page instead, since building a 

49page resolves (and caches) its resources and contents. 

50``after_page()`` estimates what the library's caches hold 

51since the last release -- decoded stream bytes, plus 

52``_BYTES_PER_PARSED_ELEMENT`` for every object the walk has parsed and 

53for every element nested in it (a stream's dictionary included, read 

54without decoding the stream: a Form XObject's can hold any number of 

55entries), including the objects pdfminer parses 

56out of object streams (each counted once, whether or not a page has 

57resolved it through the object cache), and for every font pdfminer's 

58resource manager builds or hands out and every element nested in the 

59tables a build stores on the font (``_font_tables()``; a font or a 

60table seen again is counted once, but see below), with every string 

61among all of these counted by its size, plus the decoded bytes each 

62build feeds to a parser it leaves in a reference cycle 

63(``_parser_input_bytes()``) -- and, once that reaches 

64``MAX_PDF_RETAINED_DECODED_BYTES``, releases it: decoded streams go back 

65to their encoded state (the next access decodes them again from their 

66raw bytes, so a stream a later page shares still extracts correctly), 

67and every object the walk added to the cache is dropped, to be parsed 

68again from the file if a later page needs it, and the fonts are 

69dropped (the font cache cleared, the releaser's own references let 

70go), to be built again, and the reference cycles pdfminer's parsers 

71leave are collected. Objects cached before the first ``before_page()`` 

72(the page tree, when the walk materialised it up front) are kept. 

73Honest documents stay far below the threshold and are left alone. 

74 

75That estimate counts what the cases found so far showed a page can 

76keep, and each case found later (fonts written inline, widths copied 

77per font, font header copies, the strings a CID font joins) went 

78unbounded until it was counted. So the release also has a measured 

79trigger: where the process's resident memory can be read 

80(``/proc/self/statm``, on Linux), ``after_page()`` also releases once it 

81has grown by the threshold since the last release (or since the walk 

82began), whatever the estimate says, and the pdfminer releaser runs a 

83collection in the middle of a page once it has grown by the threshold 

84since the last collection. It is the whole process's memory: growth 

85that other threads cause, or that a release cannot free (pdfminer keeps 

86every name it parses in a process-wide table), makes a release or a 

87collection that frees nothing, after which the baseline is where the 

88memory then is, so the next one waits for another threshold's worth of 

89growth. On other platforms only the estimate applies. Of the eight 

90honest documents measured (up to 500 pages) it released one, once: a 

91one-page form whose three embedded TrueType fonts grew the process by 

92~69 MB. 

93 

94Fonts are counted where they are built, not read off the font cache, 

95which misses the inline ones: the releaser wraps this document's 

96``rsrcmgr.get_font`` (an instance attribute, holding the manager and 

97the releaser weakly) and counts every font it returns, once per font 

98object, remembering the fonts weakly so that it keeps none pdfminer 

99frees. To count a table several fonts share only once, it holds the 

100tables it has counted, and lets go of those nothing else holds once 

101the ones added since it last did reach the threshold's worth, after 

102every collection and on a release (a table seen again after that is a 

103new object, and counts in full); so it can keep up to about the 

104threshold's worth of tables that pdfminer would have freed sooner, such 

105as the widths of a CID font written inline in a Form XObject, built 

106again on every ``Do``, plus those that something else held when it last 

107let go (such a table weighs once more toward the next time, so it goes 

108soon after its holder, the font being built, lets go of it). The tables 

109something else still holds it keeps, and never counts or walks again: a build stores some parts of the parsed font as 

110they are (the descriptor, a CID font's /CIDSystemInfo, the items of a 

111Type 3 font's /FontMatrix), and walking those on every ``Do`` of a form 

112with an inline font cost more than the build (a page of 100 ``Do``s of 

113a form whose inline font's descriptor holds 600,000 empty arrays took 

114~66 s of CPU, and ~130 s as a Type 3 font's /FontMatrix item, against 

115~3 s without the releaser; now ~6.3-6.7 s, the extra being the one 

116release, which parses the form again to rewind it, and its 

117collection). A nested container that another holds too is counted as 

118a table of its own, for the same reason. When it lets go of tables it 

119also asks for a full garbage collection, as it does once the 

120bytes fed to parsers since the last one, times 

121``_PARSER_BYTES_PER_INPUT_BYTE``, reach the threshold: what those 

122parsers, and the fonts pdfminer has dropped, hold in reference cycles 

123is garbage as soon as a build returns, so it is freed in the middle of 

124a page too (a page invoking a form of five inline 65,536-entry 

125/ToUnicode fonts 20 times took the process to ~880 MB before, ~110 MB 

126with it). Those bytes count only where pdfminer 

127parses them, but the factor is the worst case (a whitespace input 

128leaves nothing), and a collection costs time in proportion to every 

129object the process holds, whatever it frees. So a collection asked for 

130in the middle of a page waits until the walking thread has spent 

131``_COLLECT_SPACING`` (9) times the last collection's duration since it 

132ended -- such collections then take at most about a tenth of its CPU 

133time -- unless the resident memory has grown by ``_FORCED_GROWTH`` (4) 

134times the threshold since the last one; a release that only the 

135resident memory asks for waits the same way, and a release the 

136estimate asks for drops the caches at once but defers its collection 

137the same way (until a later font build or page, or the walk's end) -- 

138except that a collection runs at once, however recent the last one, 

139once the garbage that only a collection frees, released since the 

140last one, reaches the threshold: what those parsers were fed, which 

141each holds -- a fresh copy (a /FontFile header) at once, a stream's own 

142decoded bytes, once each, when the release rewinds the stream -- and 

143the tables of the fonts built with them (their /ToUnicode map, which 

144the parser's cycle holds) once those fonts are let go of -- an inline 

145font's when its build returns, a cached font's at the release. Spaced 

146too, as in the previous revision, those were held for as long as 

147nine full collections of the process's heap took: 40 pages of five 

148fonts sharing a 65,536-entry /ToUnicode map peaked ~400 MB above the 

149start with 8 million objects held (~430 MB without the measured 

150trigger), and now 51-74 MB, as without objects held. What a deferred 

151collection would free besides is the part of the parser residue the 

152factor counts beyond each parser's input, which stays held until it 

153runs, for longer the more objects the process holds (on Linux at most 

154until the resident memory has grown by four times the threshold); one 

155still pending when the walk ends runs then (``close()``, or once the 

156releaser is garbage), and honest documents defer none. Before the 

157spacing, a page of 

1582,000 ``Do``s of a form with one inline Type 1 font whose /FontFile 

159has a /Length1 of 0 (a 1.3 KB file) ran 2,001 collections: ~141 s of 

160CPU instead of ~0.45 s, with a million objects held; it now runs none, 

161and with a /Length1 of the whole 400 KB of whitespace (which the 

162factor over-counts) 11 instead of 2,001 (~6.2 s instead of ~191 s, 

163against ~4.7 s without the releaser). And 100 pages whose font decodes 

164a shared 64 MB /FontFile of zeros and parses none of it (a 92 KB file), 

165which releases on every page, ran 101 collections with a million 

166objects held (~25 s of CPU) and now 25 (~20 s; the rest is decoding 

167the stream again on every page). Not bounded here: the names and 

168keywords pdfminer interns, in process-wide tables 

169(``psparser.PSLiteralTable`` and ``PSKeywordTable``) that it never 

170clears and compares against by identity, so the releaser cannot drop 

171them -- a 2 MB PDF of a million distinct names or keywords adds 

172~175 MB to the process for good, and each upload of new ones adds 

173more; what 

174one font build allocates before it returns (a /W range, a bfrange or 

175a crafted parser input expands within that build), and what the fonts 

176pdfminer still holds (cached ones, a form's while it runs) keep until 

177the release after the page. 

178 

179Each ``after_page()`` scans the document's object cache, so it costs 

180time in proportion to that cache: well under a second over a whole 

181500-page honest document, ~0.16 s per page for a cache of a million 

182objects, which takes a page tree of about a million pages (itself an 

183O(all pages) cost paid before the walk). A pdfminer release also runs a 

184full garbage collection (pdfminer's parsers leave what they built in 

185reference cycles), which costs time in proportion to the objects the 

186whole process holds: ~0.06 s per million, and so does each collection 

187the font counting runs between releases; the spacing above keeps all 

188of them to about a tenth of the walking thread's CPU time (apart from 

189those a growth of four times the threshold forces, and those the 

190threshold's worth of garbage only a collection frees forces, which 

191cost builds of that size first). On the honest 

192documents measured (eight, up to 500 pages) only the one release above 

193ran. That time is spent in the walking thread, between the CPU budget's checks, and counts 

194against it. 

195 

196These are library internals, written against the pdfplumber, 

197pdfminer.six and pypdf versions ``pdm.lock`` pins. Every access is 

198guarded: if an attribute is missing or anything raises, the releaser 

199stops releasing for that document and extraction carries on 

200unchanged (except the lookups of the streams a font build read, which 

201the document's data can make fail: one that fails counts nothing, as 

202the build could not have read that stream either; after a build that 

203raised they are looked up in the object cache only, which parses 

204nothing and follows no reference cycle), and the pdfminer 

205releaser removes its ``get_font`` wrapper and lets go of the font 

206tables it holds. The first time that happens to 

207each library's releaser in 

208a process (so at most two warnings) it is logged at WARNING, since the 

209memory bound is then off; later occurrences are logged at DEBUG. 

210pdfminer's object-stream cache entries are read as the ``(objects, 

211count)`` pairs it stores, taken in insertion order; an entry of another 

212shape is counted whole if it is a container (an over-count, which 

213brings the release sooner), and otherwise switches the releaser off. The canary tests in 

214``tests/research_library/services/test_pdf_extraction_bounds.py`` 

215(``TestLibraryInternalsTheReleaseReliesOn``) fail if a library upgrade 

216removes what the releasers rely on, so an upgrade cannot undo them 

217silently. 

218""" 

219 

220from __future__ import annotations 

221 

222import gc 

223import os 

224import sys 

225import threading 

226import time 

227import weakref 

228from itertools import islice 

229from typing import Any 

230 

231from loguru import logger 

232 

233from . import pdf_extraction_limits 

234 

235# Estimated bytes a parsed object element (an array item, a dictionary 

236# entry) keeps alive. Measured on a 500,000-entry /Widths array: ~124 

237# bytes per entry in pypdf (a boxed number object plus its list slot), 

238# and in pdfminer the cached array plus one font's own widths table 

239# together, which are now each counted (an over-count, which brings the 

240# release sooner). 

241_BYTES_PER_PARSED_ELEMENT = 128 

242 

243 

244# Bytes a pdfminer parser that a font build leaves in a reference cycle 

245# can keep per byte it was fed: the tokens a ToUnicode CMap or a Type 1 

246# font header leaves on the parser's stack. Measured on pdfminer.six 

247# 20260107 (CMapParser and Type1FontHeaderParser alike): ~52 for "1 1 1", 

248# ~84 for "<<<<", ~168 for "[[[[" or "{{{{", 0 for whitespace. This is 

249# the worst case, so it over-counts most inputs (all of a whitespace 

250# one); what that over-count can cost is bounded by the spacing of the 

251# collections it asks for (_COLLECT_SPACING). 

252_PARSER_BYTES_PER_INPUT_BYTE = 192 

253 

254# A collection that the threshold asks for in the middle of a page waits 

255# until the walking thread has spent this many times the last 

256# collection's duration since it ended, so that those collections take 

257# at most about a tenth of the thread's CPU time (a full collection 

258# costs ~0.06 s per million objects the process holds, whatever it 

259# frees) -- unless the process's resident memory has grown by 

260# _FORCED_GROWTH times the threshold since the last one. 

261_COLLECT_SPACING = 9 

262_FORCED_GROWTH = 4 

263 

264# How often, in seconds of the walking thread's CPU time, a font build 

265# reads the process's resident memory (~10 us a read). 

266_RSS_CHECK_INTERVAL = 0.01 

267 

268_STRINGS = (str, bytes, bytearray) 

269 

270_rss_unavailable = False 

271 

272 

273def _resident_bytes() -> int | None: 

274 """The process's resident memory in bytes, read from 

275 ``/proc/self/statm``; None where there is no such file (not Linux) 

276 or it cannot be read.""" 

277 global _rss_unavailable 

278 if _rss_unavailable: 278 ↛ 279line 278 didn't jump to line 279 because the condition on line 278 was never true

279 return None 

280 try: 

281 with open("/proc/self/statm", "rb") as statm: 

282 pages = int(statm.read().split()[1]) 

283 return pages * os.sysconf("SC_PAGE_SIZE") 

284 except (FileNotFoundError, AttributeError, ValueError, IndexError): 

285 # Not Linux (or not a statm we know): measure nothing from now on. 

286 _rss_unavailable = True 

287 except OSError: 

288 pass 

289 return None 

290 

291 

292def _measured(previous: int | None) -> int | None: 

293 """A new resident-memory baseline: the memory now, or *previous* if 

294 it cannot be read this time (a transient failure must not leave the 

295 baseline unknown, which would switch the measured trigger off).""" 

296 now = _resident_bytes() 

297 return previous if now is None else now 

298 

299 

300def _font_tables(font: object) -> list[object]: 

301 """The values a pdfminer font build stored that ``_font_built()`` 

302 counts: every dictionary, list, tuple and string (``str`` or 

303 ``bytes``, by its size) held in an attribute of the font or of an 

304 object held in one (its ``unicode_map``, its ``cmap``), streams 

305 aside (their decoded bytes count where they are cached). 

306 

307 That is the ``/ToUnicode`` map (``cid2unichr``) and the values its 

308 ``def``s set (``attrs``), the encoding's code-to-unicode table 

309 (``cid2unicode``, a copy per font when /Differences or the embedded 

310 font program gives one), the ``widths`` (simple and CID fonts alike: 

311 each build makes its own, with every array a width points at copied, 

312 even when many fonts share one /Widths array), a CID font's 

313 ``disps``, the descriptor, the bounding box, the font matrix, and 

314 strings such as a CID font's ``cidcoding``, which each build makes 

315 by joining its /CIDSystemInfo /Registry and /Ordering, however long 

316 those strings are and however many fonts share them. Every element 

317 nested in a container counts like a parsed element (a /ToUnicode 

318 map measured ~1.05x the estimate), and every string nested in one 

319 by its size as well. A value several fonts share (pdfminer's 

320 built-in encodings and metrics, a predefined CMap, a font 

321 descriptor) is counted once, and not again while something else 

322 holds it (``PdfplumberWalkReleaser._let_go_of_tables()``). 

323 """ 

324 from pdfminer.pdftypes import PDFStream 

325 

326 counted = _CONTAINERS + _STRINGS 

327 found = [] 

328 held = [] 

329 for value in getattr(font, "__dict__", {}).values(): 

330 if isinstance(value, counted): 

331 found.append(value) 

332 elif hasattr(value, "__dict__") and not isinstance( 

333 value, (PDFStream, type) 

334 ): 

335 held.append(value) 

336 for owner in held: 

337 found.extend(v for v in vars(owner).values() if isinstance(v, counted)) 

338 return found 

339 

340 

341def _cached_value(value: object) -> object: 

342 """*value*, or, if it is a reference, what it resolves to in its 

343 document's object cache, without parsing anything: a reference that 

344 is not cached, or that leads back to one already followed (``8 0 

345 obj 8 0 R``), resolves to None.""" 

346 from pdfminer.pdftypes import PDFObjRef 

347 

348 followed = set() 

349 while isinstance(value, PDFObjRef): 

350 if value.objid in followed: 350 ↛ 351line 350 didn't jump to line 351 because the condition on line 350 was never true

351 return None 

352 followed.add(value.objid) 

353 cache = getattr(value.doc, "_cached_objs", None) 

354 entry = cache.get(value.objid) if isinstance(cache, dict) else None 

355 if not isinstance(entry, tuple) or not entry: 

356 return None 

357 value = entry[0] 

358 return value 

359 

360 

361def _parser_input_bytes( 

362 spec: object, built: bool = True, inputs: list | None = None 

363) -> int: 

364 """Bytes the build of a font from *spec* fed to a pdfminer parser 

365 that it leaves in a reference cycle, under the conditions pdfminer 

366 parses them (``PDFResourceManager.get_font()`` picks the class from 

367 ``/Subtype``): 

368 

369 - the decoded ``/ToUnicode`` CMap (``CMapParser``): resolved by a 

370 simple font (Type 1, TrueType, Type 3, or an unknown subtype); by a 

371 CID font only if it is a stream object itself rather than a 

372 reference (a Type 0 font hands its descendant the resolved one); a 

373 Type 0 font parses nothing itself, its descendant's build does; 

374 - for a Type 1 or TrueType font without ``/Encoding`` whose 

375 ``/BaseFont`` is not one of the standard fonts pdfminer has 

376 built-in metrics for (whose descriptor replaces the spec's), the 

377 copy of its decoded ``/FontFile``'s first ``/Length1`` bytes that 

378 ``Type1FontHeaderParser`` reads: exactly the length of that slice. 

379 

380 A stream counts only if it is decoded (the lookup decodes nothing), 

381 which it is after a build that parsed it; a lookup that fails counts 

382 nothing, as the build could not have read that stream either. After 

383 a build that returned (*built*), the references followed are ones 

384 the build resolved itself; after one that raised, it may not have 

385 got that far, so they are looked up in the object cache only 

386 (``_cached_value()``): what the build resolved is there, and nothing 

387 is parsed, nor followed round a cycle. 

388 

389 With *inputs*, each stream's decoded bytes the count includes are 

390 appended to it with the number of them fed, ``(data, fed)``: the 

391 parser holds *data* itself when it was fed all of it (pdfminer 

392 wraps it in a ``BytesIO``, which shares it), and a fresh copy of 

393 the first *fed* bytes otherwise.""" 

394 from pdfminer.pdffont import FontMetricsDB 

395 from pdfminer.pdftypes import PDFStream, resolve1 

396 from pdfminer.psparser import PSLiteral, literal_name 

397 

398 if not isinstance(spec, dict): 

399 return 0 

400 resolve = resolve1 if built else _cached_value 

401 

402 def decoded(stream: object) -> bytes | None: 

403 if isinstance(stream, PDFStream) and stream.data is not None: 

404 return stream.data 

405 return None 

406 

407 def resolved(holder: dict, key: str) -> object: 

408 try: 

409 return resolve(holder.get(key)) 

410 except Exception as exc: 

411 logger.debug("font {} lookup failed ({})", key, type(exc).__name__) 

412 return None 

413 

414 subtype = spec.get("Subtype") 

415 subtype = ( 

416 literal_name(subtype) 

417 if isinstance(subtype, PSLiteral) 

418 else ("Type1" if subtype is None else None) 

419 ) 

420 if subtype == "Type0": 420 ↛ 421line 420 didn't jump to line 421 because the condition on line 420 was never true

421 return 0 

422 if subtype in ("CIDFontType0", "CIDFontType2"): 

423 data = decoded(spec.get("ToUnicode")) 

424 if data is None: 424 ↛ 426line 424 didn't jump to line 426 because the condition on line 424 was always true

425 return 0 

426 if inputs is not None: 

427 inputs.append((data, len(data))) 

428 return len(data) 

429 data = decoded(resolved(spec, "ToUnicode")) 

430 total = len(data) if data is not None else 0 

431 if data is not None and inputs is not None: 

432 inputs.append((data, total)) 

433 if subtype == "Type3" or "Encoding" in spec: 

434 return total 

435 basefont = spec.get("BaseFont") 

436 if isinstance(basefont, PSLiteral): 

437 try: 

438 FontMetricsDB.get_metrics(literal_name(basefont)) 

439 return total # built-in metrics: the spec's descriptor unused 

440 except KeyError: 

441 pass 

442 except Exception: 

443 return total # the build raised before reading the font file 

444 descriptor = resolved(spec, "FontDescriptor") 

445 if not isinstance(descriptor, dict) or "FontFile" not in descriptor: 

446 return total 

447 font_file = resolved(descriptor, "FontFile") 

448 data = decoded(font_file) 

449 if data is None or "Length1" not in font_file.attrs: 

450 return total 

451 # pdfminer's int_value(): anything but an int is 0. 

452 length1 = resolved(font_file.attrs, "Length1") 

453 if not isinstance(length1, int): 453 ↛ 454line 453 didn't jump to line 454 because the condition on line 453 was never true

454 return total 

455 # The parser reads data[:length1], whatever the sign of /Length1. 

456 header = len(range(len(data))[:length1]) 

457 if header and inputs is not None: 

458 inputs.append((data, header)) 

459 return total + header 

460 

461 

462# Libraries whose releaser has already warned that it disabled itself in 

463# this process; later disables are logged at DEBUG only. 

464_warned_disabled: set[str] = set() 

465_warned_lock = threading.Lock() 

466 

467 

468def _log_disabled(library: str, exc: BaseException | None) -> None: 

469 """Log a releaser disabling itself: once per library at WARNING.""" 

470 reason = type(exc).__name__ if exc is not None else "unknown" 

471 with _warned_lock: 

472 first = library not in _warned_disabled 

473 _warned_disabled.add(library) 

474 if first: 

475 logger.warning( 

476 "{} cache release disabled ({}): memory held between PDF pages " 

477 "is no longer bounded. The library internals it relies on may " 

478 "have changed; run tests/research_library/services/" 

479 "test_pdf_extraction_bounds.py. Further occurrences are logged " 

480 "at DEBUG.", 

481 library, 

482 reason, 

483 ) 

484 else: 

485 logger.debug( 

486 "{} cache release disabled for this document ({})", 

487 library, 

488 reason, 

489 ) 

490 

491 

492def _limit() -> int: 

493 return pdf_extraction_limits.MAX_PDF_RETAINED_DECODED_BYTES 

494 

495 

496_CONTAINERS = (dict, list, tuple) 

497 

498 

499def _sole_holder_refcount() -> int: 

500 """What ``sys.getrefcount()`` reports, read the way ``_count_elements()`` 

501 and ``_let_go_of_tables()`` read it (on a loop variable), for a value 

502 that one container holds and nothing else (3 on CPython 3.12, 2 on 

503 3.14, whose loops borrow the reference); a value something else 

504 holds too reads more.""" 

505 counts = [sys.getrefcount(value) for value in [[]]] 

506 return counts[0] 

507 

508 

509_SOLE_HOLDER = _sole_holder_refcount() 

510 

511# Passes _let_go_of_tables() makes at most (see there). 

512_LET_GO_PASSES = 4 

513 

514 

515def _count_elements( 

516 obj: object, skip: tuple[type, ...], shared: list | None = None 

517) -> int: 

518 """Number of items in the lists and dicts nested in *obj*, plus, for 

519 every string (``str`` or ``bytes``) among them, among the dicts' 

520 keys, or *obj* itself, its size in units of 

521 ``_BYTES_PER_PARSED_ELEMENT`` (none for a short one, which its item 

522 already accounts for). Keys count because pdfminer turns a name that 

523 is not UTF-8 into the text of its repr, about four characters a 

524 byte. 

525 

526 Objects of the *skip* types nested in *obj* (streams, whose data is 

527 accounted by its decoded bytes, and whose dictionary is counted 

528 where the stream is cached) are not descended into; *obj* itself is 

529 counted whatever its type, so a pypdf stream, which is its own 

530 dictionary, counts its entries (a pdfminer stream's dictionary is 

531 its ``attrs``). A dictionary is read through ``dict``'s own methods, 

532 which a subclass cannot override to resolve references. Parsed PDF 

533 objects are trees, so the walk is linear in their size -- no more 

534 than parsing them cost. 

535 

536 With *shared*, a nested container that something besides its parent 

537 also holds is appended to *shared* instead of descended into, so 

538 that the caller counts it once, however many tables hold it: a font 

539 build can store a new container that holds parts of a parsed object 

540 (a Type 3 font's ``matrix`` is a new tuple of its /FontMatrix 

541 items), and walking those parts again on every build would cost more 

542 than the build did. 

543 """ 

544 count = 0 

545 stack = [obj] 

546 while stack: 

547 item = stack.pop() 

548 if item is not obj and isinstance(item, skip): 

549 continue 

550 if isinstance(item, _STRINGS): 

551 count += sys.getsizeof(item) // _BYTES_PER_PARSED_ELEMENT 

552 continue 

553 if isinstance(item, dict): 

554 items: Any = dict.values(item) 

555 keys = dict.keys(item) 

556 if any(issubclass(t, _STRINGS) for t in set(map(type, keys))): 

557 count += sum( 

558 sys.getsizeof(key) // _BYTES_PER_PARSED_ELEMENT 

559 for key in keys 

560 if isinstance(key, _STRINGS) 

561 ) 

562 elif isinstance(item, (list, tuple)): 

563 items = item 

564 else: 

565 continue 

566 count += len(item) 

567 # A flat table (a font's widths, a /ToUnicode map) is told apart 

568 # from its item types alone, without a Python-level step per item. 

569 types = set(map(type, items)) 

570 if any(issubclass(t, _CONTAINERS) for t in types): 

571 if shared is None: 

572 stack.extend(items) 

573 continue 

574 for value in items: 

575 if ( 

576 isinstance(value, _CONTAINERS) 

577 and sys.getrefcount(value) > _SOLE_HOLDER 

578 ): 

579 shared.append(value) 

580 else: 

581 stack.append(value) 

582 elif any(issubclass(t, _STRINGS) for t in types): 

583 count += sum( 

584 sys.getsizeof(value) // _BYTES_PER_PARSED_ELEMENT 

585 for value in items 

586 if isinstance(value, _STRINGS) 

587 ) 

588 return count 

589 

590 

591def _is_objstm_entry(value: object) -> bool: 

592 """Whether *value* has the shape pdfminer gives a ``_parsed_objs`` 

593 entry: ``(objects, count)``, a list of the stream's header numbers 

594 and objects plus the number of objects.""" 

595 return ( 

596 isinstance(value, tuple) 

597 and len(value) == 2 

598 and isinstance(value[0], list) 

599 and isinstance(value[1], int) 

600 ) 

601 

602 

603def _require_container(value: object) -> None: 

604 """Raise unless ``_count_elements`` can count *value*: an entry with 

605 nothing to count it by stops the releaser (``_disable()`` warns) 

606 rather than going uncounted.""" 

607 if not isinstance(value, (dict, list, tuple)): 

608 raise TypeError("unexpected pdfminer object stream cache entry") 

609 

610 

611class _Pending: 

612 """What a pdfminer walk has left for the cycle collector: whether a 

613 collection is due (asked for, and deferred by the spacing), and an 

614 estimate, in bytes, of the garbage that only a collection frees, 

615 released since the last one (``PdfplumberWalkReleaser._font_parsed()``). 

616 Kept apart from the releaser so that the walk-end collection 

617 (``_collect_at_walk_end()``) can read it once the releaser is gone.""" 

618 

619 __slots__ = ("due", "garbage") 

620 

621 def __init__(self) -> None: 

622 self.due = False 

623 self.garbage = 0 

624 

625 

626def _collect_at_walk_end(pending: _Pending) -> None: 

627 """Run the collection a walk deferred, if what it would free is 

628 garbage only a collection frees, so that the walk does not end 

629 holding it. Honest documents never ask for one.""" 

630 if pending.due and pending.garbage > 0: 

631 pending.due = False 

632 pending.garbage = 0 

633 gc.collect() 

634 

635 

636class PdfplumberWalkReleaser: 

637 """Releases pdfminer's caches while pdfplumber walks a document.""" 

638 

639 def __init__(self, pdf: Any) -> None: 

640 self._pdf = pdf 

641 self._enabled = True 

642 self._pre_walk_keys: set | None = None 

643 self._page_mark: int | None = None 

644 # Set by before_page() alongside _page_mark (keep it there when 

645 # merging other before_page() changes): without it after_page() 

646 # counts every object stream parsed before the page, including 

647 # the pre-walk ones, against the threshold. 

648 self._objstm_mark: int | None = None 

649 self._parsed_elements = 0 

650 # Fonts counted since the last release, by id, held weakly: a 

651 # font handed out again (from pdfminer's cache, or as a Type0 

652 # font's descendant) is not counted twice, and holding the fonts 

653 # would keep ones pdfminer itself lets go of -- every unknown 

654 # font name in a Tf builds a new font that nothing caches. 

655 self._fonts_seen: weakref.WeakValueDictionary[int, Any] = ( 

656 weakref.WeakValueDictionary() 

657 ) 

658 # The font tables and long strings counted (by id, and held so 

659 # that no id is reused for a new one while it is here), so one 

660 # that several fonts share counts once; and the entries of those 

661 # added since the tables were last let go of (at least one per 

662 # table, so that empty tables cannot pile up). Holding them 

663 # keeps values pdfminer may already have let go of (an inline 

664 # CID font's widths die with the font), so once those entries 

665 # reach the threshold, and after every collection, 

666 # _let_go_of_tables() lets go of the tables nothing else holds; 

667 # the rest (a descriptor or a /FontMatrix item that a parsed 

668 # object holds, pdfminer's built-in tables) cost nothing to hold 

669 # and are kept, so that they are never walked again. 

670 self._table_ids: set[int] = set() 

671 self._tables: list[object] = [] 

672 self._held_entries = 0 

673 # The entries of each table added since the tables were last 

674 # let go of, by id: one kept then (something else still held 

675 # it, such as the font being built) weighs again until the next 

676 # time, so that it is let go of soon after that holder is. 

677 self._new_tables: dict[int, int] = {} 

678 # Decoded bytes the font builds since the last release fed to 

679 # parsers left in reference cycles (_parser_input_bytes()), 

680 # counted against the threshold; and since the last collection 

681 # of those cycles. 

682 self._font_bytes = 0 

683 self._uncollected_bytes = 0 

684 # The measured side (_resident_bytes(); None where it cannot be 

685 # read): the process's resident memory after the last release 

686 # and after the last collection (or when the walk began), and 

687 # when (in the thread's CPU time) the last collection ended and 

688 # how long it took, which space the next one (_may_collect()). 

689 self._release_rss: int | None = None 

690 self._collect_rss: int | None = None 

691 self._collect_end = 0.0 

692 self._collect_cost = 0.0 

693 # Whether a collection is due, and the garbage only a collection 

694 # frees, released since the last one: the decoded bytes fed to 

695 # parsers (which those parsers keep in reference cycles once the 

696 # build returns) and the tables of the fonts built with them 

697 # (their /ToUnicode map, which the parser's cycle holds) once 

698 # those fonts are let go of -- an inline font's when its build 

699 # returns, a cached font's at the next release, when the font 

700 # cache is cleared. A collection runs at once, whatever the 

701 # spacing, when that garbage reaches the threshold 

702 # (_font_parsed(), after_page()). The tables of a build not 

703 # classified yet (a Type0 font's descendant, built inside it) 

704 # and of the cached fonts built since the last release: 

705 self._pending = _Pending() 

706 self._unclassified = 0 

707 self._cached_garbage = 0 

708 # The decoded streams fed whole to parsers since the last 

709 # release (their sizes, by the id of the bytes, which the stream 

710 # holds until the release rewinds it): garbage that only a 

711 # collection frees from then on. 

712 self._fed_streams: dict[int, int] = {} 

713 # A deferred collection still pending when the walk ends runs 

714 # then, when the releaser goes (or on close()). 

715 self._walk_end = weakref.finalize( 

716 self, _collect_at_walk_end, self._pending 

717 ) 

718 self._walk_end.atexit = False 

719 # A release whose collection was deferred: the collection that 

720 # follows sets _release_rss. 

721 self._release_collect_due = False 

722 self._rss_checked = 0.0 

723 self._rsrcmgr: Any = None 

724 # ids of the objects counted from object-stream lists since the 

725 # last release. Every one stays reachable from _parsed_objs until 

726 # that release clears the set, so no id is reused for a different 

727 # object while it is here. 

728 self._counted: set[int] = set() 

729 

730 def _count_fonts(self) -> None: 

731 """Wrap this document's ``rsrcmgr.get_font`` to count every font 

732 it returns, where it is built. 

733 

734 pdfminer caches only fonts that are objects of their own 

735 (``_cached_fonts``, keyed by object number); a font dictionary 

736 written inline in a ``/Resources /Font`` dictionary -- a page's, 

737 or a Form XObject's, initialised on every ``Do`` -- is built again 

738 each time and cached nowhere, so the cache cannot show what a 

739 page built. Every font goes through ``get_font()`` 

740 (``init_resources()``, an unknown font name in ``Tf``, and a 

741 Type0 font's descendant), which pdfminer calls on the instance. 

742 The wrapper is an instance attribute, so only this document's 

743 resource manager is affected, and it holds the manager and the 

744 releaser weakly: it adds no reference cycle, and once the 

745 releaser is gone or disabled it only passes calls through. 

746 """ 

747 from pdfminer.pdfinterp import PDFResourceManager 

748 

749 rsrcmgr = self._pdf.rsrcmgr 

750 if not isinstance(rsrcmgr, PDFResourceManager) or "get_font" in vars( 750 ↛ 753line 750 didn't jump to line 753 because the condition on line 750 was never true

751 rsrcmgr 

752 ): 

753 raise TypeError("unexpected pdfminer resource manager") 

754 cached = rsrcmgr._cached_fonts 

755 if not isinstance(cached, dict): 755 ↛ 756line 755 didn't jump to line 756 because the condition on line 755 was never true

756 raise TypeError("unexpected pdfminer font cache") 

757 build = type(rsrcmgr).get_font 

758 manager = weakref.ref(rsrcmgr) 

759 releaser = weakref.ref(self) 

760 # How many get_font() calls are running: a Type0 font builds its 

761 # descendant with a nested one, which is not cached, but lives 

762 # as long as the Type0 font does. 

763 depth = [0] 

764 

765 def get_font(objid: object, spec: Any) -> Any: 

766 depth[0] += 1 

767 try: 

768 font = build(manager(), objid, spec) 

769 except BaseException: 

770 # A build that fails part-way may have parsed its 

771 # /ToUnicode already. 

772 owner = releaser() 

773 if owner is not None: 773 ↛ 775line 773 didn't jump to line 775 because the condition on line 773 was always true

774 owner._font_parsed(spec, built=False, outer=depth[0] == 1) 

775 raise 

776 finally: 

777 depth[0] -= 1 

778 owner = releaser() 

779 if owner is not None: 

780 fonts = getattr(manager(), "_cached_fonts", None) 

781 owner._font_built( 

782 font, 

783 spec, 

784 outer=depth[0] == 0, 

785 cached=bool(objid) 

786 and isinstance(fonts, dict) 

787 and fonts.get(objid) is font, 

788 ) 

789 return font 

790 

791 # Fonts cached before the walk are not its cost. 

792 for font in cached.values(): 

793 self._fonts_seen[id(font)] = font 

794 rsrcmgr.get_font = get_font 

795 self._rsrcmgr = rsrcmgr 

796 

797 def _font_built( 

798 self, 

799 font: object, 

800 spec: object, 

801 outer: bool = True, 

802 cached: bool = False, 

803 ) -> None: 

804 """Count a font ``get_font()`` returned: one element, the 

805 elements of its tables not counted yet, and the bytes its build 

806 fed to parsers (``_font_parsed()``). A font handed out again 

807 (from the cache, or a Type0 font's descendant) was not built 

808 again. A table, or a container nested in one, that something 

809 else holds too is counted once, as a table of its own, and kept 

810 while that holder keeps it (``_let_go_of_tables()``): a build 

811 stores some parts of the parsed font as they are (the 

812 descriptor, a CID font's /CIDSystemInfo, the items of a Type 3 

813 font's /FontMatrix), and an inline font in a form is built again 

814 on every ``Do``, so walking them again would cost more than the 

815 build (a 600,000-entry descriptor took ~0.9 s a ``Do``). *outer* 

816 is False for a build nested in another (a Type0 font's 

817 descendant), and *cached* tells whether pdfminer cached the font 

818 (``_font_parsed()``). Never raises (it runs inside pdfminer's 

819 page interpretation).""" 

820 if not self._enabled: 820 ↛ 821line 820 didn't jump to line 821 because the condition on line 820 was never true

821 return 

822 try: 

823 from pdfminer.pdftypes import PDFStream 

824 

825 if self._fonts_seen.get(id(font)) is font: 

826 if outer and self._unclassified: 826 ↛ 827line 826 didn't jump to line 827 because the condition on line 826 was never true

827 self._font_parsed(None, outer=True, cached=cached) 

828 return 

829 self._fonts_seen[id(font)] = font 

830 self._parsed_elements += 1 

831 built = 0 

832 work = _font_tables(font) 

833 while work: 

834 table = work.pop() 

835 if id(table) in self._table_ids: 

836 continue 

837 shared: list = [] 

838 entries = _count_elements(table, (PDFStream,), shared) 

839 if isinstance(table, _STRINGS) and not entries: 

840 continue # short: the font's own element covers it 

841 self._table_ids.add(id(table)) 

842 self._tables.append(table) 

843 self._held_entries += max(1, entries) 

844 self._new_tables[id(table)] = max(1, entries) 

845 self._parsed_elements += entries 

846 built += entries 

847 work.extend(shared) 

848 let_go = self._held_entries * _BYTES_PER_PARSED_ELEMENT >= _limit() 

849 except Exception as exc: 

850 self._disable(exc) 

851 return 

852 # What the fonts pdfminer has let go of (an inline font built on 

853 # every Do of a form) holds in reference cycles is now free to 

854 # collect, in the middle of the page; the collection lets go of 

855 # the tables that leaves only the releaser holding. If the 

856 # spacing defers it, they are let go of here all the same. 

857 collected = self._collect_end 

858 self._font_parsed( 

859 spec, collect=let_go, entries=built, outer=outer, cached=cached 

860 ) 

861 if let_go and self._enabled and self._collect_end == collected: 

862 try: 

863 self._let_go_of_tables() 

864 except Exception as exc: 

865 self._disable(exc) 

866 

867 def _let_go_of_tables(self) -> None: 

868 """Let go of the font tables held that nothing else holds (the 

869 ones pdfminer has let go of), and keep the others: holding them 

870 costs nothing, their ids stay theirs, and so they are not 

871 counted, nor walked, again. Called once the entries of the 

872 tables added since the last call reach the threshold's worth, 

873 after every collection (which frees the reference cycles that 

874 hold the tables of fonts pdfminer dropped) and on a release.""" 

875 tables, self._tables = self._tables, [] 

876 # A container only the list holds, read the way the loop below 

877 # reads the tables: if it reads more (an interpreter whose 

878 # references differ from the calibration), the counts cannot be 

879 # trusted, and every table is let go of -- counted again if seen 

880 # again, an over-count -- rather than kept while nothing holds it. 

881 probe = [[]] 

882 trusted = True 

883 for index in range(len(probe)): 

884 table = probe[index] 

885 trusted = sys.getrefcount(table) <= _SOLE_HOLDER 

886 table = probe = None 

887 if not trusted: 

888 tables = [] 

889 # Each one let go of at once, so that a table held only by 

890 # another one (a nested container counted as a table of its own) 

891 # goes with it; one counted before its holder goes in the next 

892 # pass (pdfminer's builds nest them a level or two deep). 

893 kept: list = [] 

894 for _ in range(_LET_GO_PASSES): 894 ↛ 906line 894 didn't jump to line 906 because the loop on line 894 didn't complete

895 kept = [] 

896 for index in range(len(tables)): 

897 table = tables[index] 

898 if sys.getrefcount(table) > _SOLE_HOLDER: 

899 kept.append(table) 

900 else: 

901 tables[index] = None 

902 table = None 

903 if len(kept) == len(tables): 

904 break 

905 tables = kept 

906 self._tables = kept 

907 self._table_ids = {id(table) for table in kept} 

908 # A table kept for the first time weighs once more: what still 

909 # held it may be about to let go of it (the font being built, a 

910 # cycle not collected yet). One kept again weighs nothing. 

911 new, self._new_tables = self._new_tables, {} 

912 self._held_entries = sum(new.get(id(table), 0) for table in kept) 

913 

914 def _font_parsed( 

915 self, 

916 spec: object, 

917 collect: bool = False, 

918 built: bool = True, 

919 entries: int = 0, 

920 outer: bool = True, 

921 cached: bool = False, 

922 ) -> None: 

923 """Count the bytes a font build from *spec* fed to parsers that 

924 pdfminer leaves in reference cycles, and collect those cycles 

925 once what they can hold reaches the threshold (or when 

926 *collect*), or once the process's resident memory has grown by 

927 the threshold since the last collection -- spaced by 

928 ``_may_collect()`` either way. 

929 

930 The parser and what it was fed are garbage as soon as the build 

931 returns, but only the cycle collector frees them, and they can 

932 create too few tracked objects for it to run: a Type 1 font 

933 without /Encoding parses a fresh copy of its /FontFile's first 

934 /Length1 bytes, and three such fonts a page sharing a 20 MB 

935 /FontFile of spaces grew ~57 MB a page; the tokens a crafted 

936 /ToUnicode leaves on its parser's stack hold up to 

937 ``_PARSER_BYTES_PER_INPUT_BYTE`` times its size. Collecting here, 

938 not after the page, also bounds a page that builds many such 

939 fonts. *built* is False after a build that raised 

940 (``_parser_input_bytes()`` then parses nothing). 

941 

942 Those collections are spaced (``_may_collect()``), because the 

943 factor is the worst case, except once the garbage that only a 

944 collection frees, released since the last one, reaches the 

945 threshold: what the parsers were fed, which each holds in its 

946 cycle -- a fresh copy (a /FontFile header) at once, a stream's 

947 own decoded bytes (which the stream holds too, and which several 

948 builds can share) once each, when the release rewinds the 

949 stream -- and the tables (*entries*, counted by 

950 ``_font_built()``) of a font whose build fed one, whose 

951 /ToUnicode map its parser's cycle holds, once the font is let go 

952 of: an inline font's (not *cached*) as soon as its outermost 

953 build (*outer*) returns, a cached font's at the next release. 

954 That collection runs at once: the garbage it frees cost 

955 builds of its size, and deferring it held what it frees for as 

956 long as a collection of the whole process's heap took, nine 

957 times over (40 pages of five fonts sharing a 65,536-entry 

958 /ToUnicode map, with 8 million objects held, peaked ~400 MB 

959 above the start). Never raises.""" 

960 if not self._enabled: 960 ↛ 961line 960 didn't jump to line 961 because the condition on line 960 was never true

961 return 

962 try: 

963 pending = self._pending 

964 inputs: list = [] 

965 fed = ( 

966 _parser_input_bytes(spec, built, inputs) 

967 if spec is not None 

968 else 0 

969 ) 

970 self._font_bytes += fed 

971 self._uncollected_bytes += fed 

972 for data, count in inputs: 

973 if count < len(data): 

974 # A fresh copy (a /FontFile header): only the 

975 # parser's cycle holds it. 

976 pending.garbage += count 

977 else: 

978 # The stream's own decoded bytes: the parser's cycle 

979 # holds them once the release rewinds the stream. 

980 self._fed_streams[id(data)] = count 

981 del inputs 

982 if fed: 

983 self._unclassified += entries * _BYTES_PER_PARSED_ELEMENT 

984 if outer: 984 ↛ 990line 984 didn't jump to line 990 because the condition on line 984 was always true

985 if cached: