Coverage for src/local_deep_research/utilities/pdf_page_walk.py: 92%

1"""Walk a pdfplumber document's pages lazily. 

2 

3pdfplumber's ``PDF.pages`` builds a ``Page`` for every page of the 

4document on first access, and ``PDF.close()`` -- which leaving a 

5``with pdfplumber.open(...)`` block calls -- accesses ``PDF.pages`` too. 

6A small upload whose page tree declares 100,000 pages therefore costs 

7O(declared pages) time and memory before any page ceiling is consulted 

8(#6771). The upload validator (``security/file_upload_validator``) 

9avoids that by walking pdfminer's page tree with 

10``PDFPage.create_pages``, which yields one page at a time; this module 

11does the same for the text-extraction walks, so a walk that stops at 

12``MAX_PDF_EXTRACTION_PAGES`` builds at most that many pages (plus the 

13one that tells it there are more). 

14 

15Callers that walk this way must not use ``with pdfplumber.open(...)`` 

16nor call ``PDF.close()``, which would build every page anyway: they 

17open the stream themselves, close it themselves, and close each page 

18they were given. 

19 

20A walk stopped early knows only a lower bound on the page count. The 

21page tree's declared ``/Count`` is deliberately not read to fill the 

22gap: it is normally an indirect reference, and resolving one can loop 

23inside pdfminer with no limit a caller can impose (a self-reference, 

24an object-stream entry whose stream is a self-reference, a stream whose 

25``/Length`` is one), outside the extraction's CPU budget. 

26""" 

27 

28from __future__ import annotations 

29 

30from typing import TYPE_CHECKING, Any, Iterator, Optional 

31 

32if TYPE_CHECKING: 

33 from .pdf_stream_release import PdfplumberWalkReleaser 

34 

35 

36def iter_pdfplumber_pages( 

37 pdf: Any, releaser: Optional[PdfplumberWalkReleaser] = None 

38) -> Iterator[Any]: 

39 """Yield *pdf*'s pages as pdfplumber ``Page`` objects, one at a time. 

40 

41 The page tree is walked only as far as the caller iterates; each 

42 ``Page`` is built exactly as ``PDF.pages`` builds it (same page 

43 number and running ``initial_doctop``). An error the page-tree walk 

44 raises is re-raised as ``PdfminerException``, as ``PDF.pages`` does. 

45 

46 Building a page resolves its ``/Resources``, ``/Contents`` and the 

47 page-tree nodes above it, which caches those objects on the 

48 document. When a *releaser* is given, its ``before_page()`` is 

49 called here, just before each page is built, so what building the 

50 page caches is inside its window and counted by the ``after_page()`` 

51 the caller makes once done with the page. The caller then must not 

52 call ``before_page()`` itself: that would move the window's start 

53 past the page's construction. 

54 """ 

55 from pdfminer.pdfpage import PDFPage 

56 from pdfplumber.page import Page 

57 from pdfplumber.utils.exceptions import PdfminerException 

58 

59 generator = PDFPage.create_pages(pdf.doc) 

60 doctop = 0 

61 page_number = 0 

62 while True: 

63 if releaser is not None: 

64 releaser.before_page() 

65 try: 

66 page_obj = next(generator) 

67 except StopIteration: 

68 return 

69 except Exception as exc: 

70 raise PdfminerException(exc) from exc 

71 page_number += 1 

72 page = Page( 

73 pdf, page_obj, page_number=page_number, initial_doctop=doctop 

74 ) 

75 doctop += page.height 

76 yield page