Coverage for src/local_deep_research/utilities/pdf_extraction_limits.py: 100%

1"""Shared ceilings for PDF text extraction. 

2 

3The storage/upload caps bound a document's file SIZE, not the COST of 

4extracting text from it: a PDF built from millions of cheap pages, or one 

5whose streams decompress into huge text, can still burn unbounded CPU 

6and memory inside the server process while a page-walking extractor 

7chews through it. 

8 

9At this tree these ceilings bound four extraction paths: 

10 

11- the download service's ``_extract_text_from_pdf`` 

12 (``research_library/services/download_service.py``, pdfplumber and its 

13 PyPDF fallback); 

14- the downloader base helper ``extract_text_from_pdf`` 

15 (``research_library/downloaders/base.py``); 

16- the upload-path extraction service 

17 (``web/services/pdf_extraction_service.py``, pdfplumber). 

18- the byte-backed document loader used by RAG uploads and Zotero sync 

19 (``document_loaders/bounded_pdf.py``, pypdf). 

20 

21On all four paths they bound exactly four things: 

22 

23- pages walked: text is extracted from at most 

24 ``MAX_PDF_EXTRACTION_PAGES`` (500) pages per attempt. The download 

25 service's PyPDF fallback runs only when pdfplumber produced no text, 

26 so one extraction there can walk up to twice that many pages (500 

27 empty pdfplumber pages, then up to 500 PyPDF pages). A longer 

28 document yields the text of at most its first 500 pages, and fewer 

29 if another ceiling stops the walk first; 

30- characters kept: at most ``MAX_PDF_EXTRACTED_CHARS`` (10,000,000) 

31 characters -- Python ``str`` characters, i.e. code points, not bytes 

32 -- including the separators that join pages; the page that crosses 

33 the ceiling contributes the prefix that fits; 

34- CPU time between pages: one whole extraction, every attempt of it 

35 included, stops starting new pages once the extracting thread has 

36 spent ``MAX_PDF_EXTRACTION_CPU_SECONDS`` of its OWN CPU time 

37 (``time.thread_time()``) since it began. Wall-clock time is 

38 deliberately not used: time spent waiting for the GIL or for other 

39 requests' extractions is not work done on this document, and counting 

40 it would let server load cut honest documents short; 

41- what the PDF library keeps between pages: pdfminer and pypdf keep 

42 every stream they decode and every object they parse for the life of 

43 the document, and closing a pdfplumber page does not release them. 

44 Once the decoded stream bytes plus an estimate of the parsed objects 

45 the walk has cached reach ``MAX_PDF_RETAINED_DECODED_BYTES`` (64 MiB), 

46 ``utilities/pdf_stream_release`` returns the decoded streams to their 

47 encoded state and drops the parsed objects (and pdfminer's fonts) the 

48 walk added. What the walk holds between pages therefore stays near 

49 that threshold instead of growing with the pages walked. 

50 

51What they do NOT bound: 

52 

53- page-tree materialisation on paths that still use eager page access. 

54 ``pypdf`` flattens the whole page tree, and accessing 

55 ``pdfplumber.PDF.pages`` builds every page object before the first page 

56 is yielded. These operations can take O(all pages) time and memory, 

57 regardless of the page ceiling. The download service and upload-path 

58 extraction instead use ``utilities/pdf_page_walk`` to walk pdfplumber 

59 pages lazily, building only the pages reached by the walk (plus the 

60 one that establishes that more pages exist). The PyPDF fallback remains 

61 eager, so its page-tree construction is not bounded by the extraction 

62 ceilings. 

63- the cost of a single page. One page's ``extract_text()`` runs to 

64 completion inside the library with no check point available to us; a 

65 few kilobytes of compressed content streams can take minutes or 

66 exhaust gigabytes of address space there, and what that page decodes 

67 and parses is held until the release after it. 

68- the exact size of what is held between releases. Parsed objects are 

69 estimated at a fixed number of bytes per array item or dictionary 

70 entry, and pdfminer's fonts keep their own copy of their widths, so 

71 the memory held can exceed the threshold by a small multiple: walks 

72 that held ~20 MB (decoded streams) or ~60 MB (parsed ``/Widths`` 

73 arrays) more per page peaked ~100-250 MB above their starting RSS. 

74 Objects cached before the walk began (the page tree, where it was 

75 built before the walk) are kept, and a 

76 released stream that a later page uses again is decoded again, at the 

77 CPU budget's expense. 

78 

79Interrupting a parse or a page that is already running would need a 

80separate process, which these constants deliberately do not introduce. 

81Text cut short by any of these ceilings is returned, and stored, like a 

82complete extraction. The upload-path service also returns a truncation flag 

83to its API and research form; the byte-backed loader marks its page documents 

84as truncated, but the RAG/Zotero callers only consume joined text, so they 

85currently expose the partial result through a logged warning. 

86""" 

87 

88from __future__ import annotations 

89 

90# 500 pages comfortably covers book-length documents; real research papers 

91# are two orders of magnitude below it. 

92MAX_PDF_EXTRACTION_PAGES = 500 

93 

94# 10 MB of text is ~5 million words — far beyond any real paper, while 

95# capping decompression-bomb output. 

96MAX_PDF_EXTRACTED_CHARS = 10_000_000 

97 

98# CPU budget, in seconds of the extracting thread's own CPU time 

99# (``time.thread_time()``), for one whole extraction: every attempt of 

100# it, from entry to return, shares this one budget. It is checked 

101# BETWEEN pages only, so it cannot interrupt a page already inside 

102# pdfplumber/pypdf, nor the parser's page-tree construction before the 

103# first page. 

104# 

105# Sized so an honest document at the page ceiling is not cut. Measured 

106# with pdfplumber 0.11.10 on CPython 3.14: a synthetic 500-page PDF of 

107# very dense text (~12,700 characters per page, about twice a dense book 

108# page) costs ~146 s of the thread's CPU when extracted alone, and 

109# ~0.5-0.66 s per page (~250-330 s for 500 pages) with 4 to 16 

110# extractions contending for the GIL, whose switching overhead 

111# thread_time() does count; an ordinary text page (~5,700 characters) 

112# costs ~0.14 s alone. 600 s leaves 2x margin over the dense document 

113# with 4 extractions contending and ~1.8x with 16. Per character that is 

114# ~23 us alone, ~39 us at 4-way and ~52 us at 16-way contention, so the 

115# character ceiling (10,000,000 characters: ~230-520 s) binds before 

116# this budget on honest text. Heavier contention, or a host several times 

117# slower, can still reach it on a dense 500-page document; the text is 

118# then cut the same way the page ceiling cuts it. 

119# 

120# The budget still stops a document whose individual pages are 

121# pathologically expensive, but it is not a hard cap on the extraction's 

122# CPU. Unchecked are: the page-tree parse before the first check 

123# (``pdfplumber.open``/``PdfReader``), the page already running when the 

124# budget runs out, and -- in the download service, when pdfplumber 

125# produced no text with budget still left -- the PyPDF fallback's 

126# ``PdfReader`` parse, which starts after the last check and runs to 

127# completion before its first page is checked. 

128MAX_PDF_EXTRACTION_CPU_SECONDS = 600.0 

129 

130# What the PDF library may keep cached between pages, in (estimated) 

131# bytes. pdfminer and pypdf keep every stream they decode, and every 

132# object they parse, for the life of the document, and page.close() does 

133# not release them: a page whose content stream is 20 MB of spaces 

134# compresses to ~20 KB, yields no characters and costs ~0.17 s of CPU, 

135# so a walk to the page ceiling held ~10 GB from a ~10 MB file; a page 

136# whose font points at a 500,000-entry /Widths array held ~60 MB more per 

137# page. Once the decoded stream bytes plus an estimate of the parsed 

138# objects the walk has cached reach this many bytes, 

139# ``utilities/pdf_stream_release`` releases them (checked between pages, 

140# like the other ceilings). Honest documents -- content streams of tens 

141# to hundreds of KB per page, a few embedded fonts -- stay far below it: 

142# no release ran on any of the real and synthetic documents measured. 

143MAX_PDF_RETAINED_DECODED_BYTES = 64 * 1024 * 1024