Develop Reference

Extract Text in Natural reading order using pymupdf (fitz)

I am trying to extract the text using pymupdf or flitz by applying this tutorial https://towardsdatascience.com/extracting-headers-and-paragraphs-from-pdf-using-pymupdf-676e8421c467
instead of
blocks = page.getText("dict")["blocks"]
I wrote
blocks = page.get_text("dict", sort=True)["blocks"]
according to https://pymupdf.readthedocs.io/en/latest/recipes-text.html
But still, the text is not in the order I expect. The first paragraph will appear in the middle.
This happens when a page has more than one column of text.

You made a good first step using the sort argument. But please note that PDF can address each single character separately, such that every basic sorting approach may fail with the "right" PDF counter example.
If a page contains n text characters, then there exist n! different ways to encode the page - all of them looking identical, but only one of them extracting the "natural" reading sequence right away.
If your page contains tables, or if the text is organized in multiple columns (as is customary in newspapers), then you must invest additional logic to cope with that.
If you use the PyMuPDF module, you can extract text in a layout preserving manner: python -m fitz gettext -mode layout ....
If you need to achieve a similar effect within your script, you may be forced to use text extraction detailed down to each single character: page.get_text("rawdict") and use the returned character positions to bring them in the right sequence.
BTW the sort parameter causes the text blocks to be sorted ascending by (1) vertical, (2) horizontal coordinate of their bounding boxes. So if in a multi-column page the second column has a slightly higher y-coordinate, it will come before the first column. To handle such a case you must use this knowledge for making specialized code.
Assuming you know have a 2-column page, then the following code snippet might be used:
width2 = page.rect.width / 2 # half of the page width
left = page.rect + (0, 0, -width2, 0) # the left half page
right = page.rect + (width2, 0, 0, 0) # the right half page
# now extract the 2 halves spearately:
lblocks = page.get_text("dict", clip=left, sort=True)["blocks"]
rblocks = page.get_text("dict", clip=right, sort=True)["blocks"]
blocks = lblocks + rblocks
# now process 'blocks'
...

How to extract text with strikethroughs from PDF files using Python

I'm currently trying to extract information from lots of PDF forms such as this:
The text 'female' should be extracted here. So contrary to my title, I'm actually trying to extract text with no strikethroughs rather than text that with strikethroughs. But if I can identify which words with strikethroughs, I can easily identify the inverse.
Gaining inspiration from this post, I came up with this set of codes:
import os
import glob
from pdf2docx import parse
from docx import Document
lst = []
files = glob.glob(os.getcwd() + r'\PDFs\*.pdf')
for i in range(len(files)):
filename = files[i].split('\\')[-1].split('.')[-2]
parse(files[i])
document = Document(os.getcwd() + rf'\PDFs\{filename}.docx')
for p in document.paragraphs:
for run in p.runs:
if run.font.strike:
lst.append(run.text)
os.remove(os.getcwd() + rf'\PDFs\{filename}.docx')
What the above code does is to convert all my PDF files into word documents (docx), and then search through the word documents for text with strikethroughs, extract those text, then delete the word document.
As you may have rightfully suspected, this set of code is very slow and inefficient, taking about 30s to run on my sample set of 4 PDFs with less than 10 pages combined.
I don't believe this is the best way to do this. However, when I did some research online, pdf2docx extracts data from PDFs using PyMuPDF, but yet PyMuPDF do not come with the capability to recognise strikethroughs in PDF text. How could this be so? When pdf2docx could perfectly convert strikethroughs in PDFs into docx document, indicating that the strikethroughs are being recognised at some level.
All in all, I would like to seek advice on whether or not it is possible to extract text with strikethroughs in PDF using Python. Thank you!

If these strikethroughs in fact are annotations, PyMuPDF offers a simple and extremely fast solution:
On a page make a list of all strikethrough annotation rectangles and extract the text "underneath" them.
Or, similarly, look at keywords you are interested in (like "male", "female") and look if any is covered by a strikethrough annot.
# strike out annotation rectangles
st_rects = [a.rect for a in page.annots(types=[fitz.PDF_ANNOT_STRIKE_OUT])]
words = page.get_text("words") # the words on the page
for rect in st_rects:
for w in words:
wrect = fitz.Rect(w[:4]) # rect of the word
wtext = w[4] # word text
if wrect.intersects(rect):
print(f"{wtext} is strike out")
# the above checks if a word area intersects a strike out rect
# B/O mostly sloppy strike out rectangle definitions the safest way.
# alternatively, simpler:
for rect in st_rects:
print(page.get_textbox(rect + (-5, -5, 5, 5)), "is striked out")
# here I have increased the strike out rect by 5 points in every direction
# in the hope to cover the respective text.
Another case are PDF drawings, so-called "line art". These are no annotations (which can be removed) but things like lines, curves, rectangles - permanently stored in the page's rendering code objects (/Contents).
PyMuPDF also lets you extract this line art. If your text is striked-out with this method, then there exist overlaps between text rectangles and line art rectangles.
Office software (MS Word, LibreOffice) usually uses thin rectangles instead of true lines to better cope with zoomed displays -- so to catch all those cases, you must select both, horizontal lines and rectangles with small absolute heights where the width is also much larger.
Here is code that extracts those horizontal lines and "pseudo-lines" and a page:
lines = [] # to be filled with horizontal "lines": thin rectangles
paths = page.get_drawings() # list of drawing dictionary objects
for path in paths: # dictionary with single draw commands
for item in path["items"]: # check item types
if item[0] in ("c", "qu"): # skip curves and quads
continue
if item[0] == "l": # a true line
p1, p2 = item[1:] # start / stop points
if p1.y != p2.y: # skip non-horizontal lines
continue
# make a thin rectangle of height 2
rect = fitz.Rect(p1.x, p1.y - 1, p2.x, p2.y + 1)
lines.append(rect)
elif item[0] == "re": # a rectangle, check if roughly a horizontal line
rect = item[1] # the item's rectangle
if rect.width <= 2 * rect.height or rect.height > 4:
continue # not a pseudo-line
lines.append(rect)
Now you can use these line rectangles to check any intersections with text rectangles.

Disclaimer: I am the author of borb, the library suggested in this answer
Ultimately, the exact code will end up varying depending on how strikethrough is implemented in your PDF. Allow me to clarify:
A PDF document (typically) has no notion of structure. So while we may see a paragraph of text, made up of several lines of text, a PDF (for the most part) just contains rendering instructions.
Things like:
Go to X, Y
Set the current font to Helvetica-Bold
Set the current color to black
Draw the letter "H"
Go to X, Y (moving slightly to the right this time)
Draw the letter "e"
etc
So in all likelihood, the text that is striked through is not marked as such in any meaningful way.
I think there are 2 options:
PDF has the concept of annotations. These are typically pieces of content that are added on top of a page. These can be extra text, geometric figures, etc. There is a specific annotation for strikethrough.
It might be an annotation, but a geometric figure (in this case a line) that simply appears over the text.
It might be a drawing instruction (inside the page content stream that is) that simply renders a black line over the text.
Your PDF might contain one (or more) of these, depending on which software initially created the strikethrough.
You can identify all of these using borb.
What I would do (in pseudo-code):
Extend SimpleTextExtraction (this is the main class in borb that deals with extracting text from a PDF)
Whenever this class sees an event (this is typically the parser having finished a particular instruction) you can check whether you saw a text-rendering instruction, or a line-drawing instruction. Keep track of text, and keep track of lines (in particular their bounding boxes).
When you have finished processing all events on a page, get all the annotations from the page, and filter out strikethrough annotations. Keep track of their bounding boxes.
From the list of TextRenderEvent objects, filter out those whose bounding box overlaps with: either a line, or a strikethrough bounding box
Copy the base algorithm for rebuilding text from these events

Vertical alignment in SVG via svgwrite

I'm having issues with vertical alignment of texts when using the svgwrite python library.
For example, here's a simple code that generates a red-filled circle with a black text saying "Text" on top of the circle:
import svgwrite
d = svgwrite.Drawing(filename='alignment_test.svg',
size=(60,60))
circle = d.circle((30,30), 30, fill='red')
text = d.text('Test', (30,30),
style='text-anchor:middle',
font_size='17px')
d.add(circle)
d.add(text)
d.save()
The result is this expected image:
However, I want the text to also be vertically aligned.
I tried using the alignment-baseline attribute of SVG, i.e.
style='text-anchor:middle;\
alignment-baseline:middle'
However, it doesn't work - neither in FireFox (where the vertical alignment just doesn't work), not in Inkscape, which complains Unimplemented style property 363.
I would appreciate any suggestions on how to solve the matter.

...so while writing this question I searched around a bit more, and actually read the second answer here (by toutankh):
vertical alignment of text element in SVG,
which suggests using the attribute dominant-baseline instead of alignment-baseline for text objects. And it works perfectly.
Lesson learned: never read only the first answer to a SO question.

How to easily place/accommodate text annotations at the edge of a plot?

I am plotting some points on a line in python using matplotlib, and whenever the point is at/near the boundaries of the plot the annotated text is hard to read due to overlapping axes labels and such (see screenshot below):
I'm currently using code like this to place my point annotations manually:
# add value text to x, y point
jt = x_points_to_plot # a single x-value, in this case
f = ys_func(x_points_to_plot) # a single y-value, in this case
ax.annotate(
'({}C, {:0.0f}%)'.format(jt, f), # the string text to add
xy=(jt + 1, f + 5), # offset the text from the point manually
ha='center')
Usually my points are in the middle and look acceptable, like this:
But I don't want to manually adjust the text for every point, because I have a lot of changing data and it's not where I want to spend my time; instead, I'd love to find a way to accommodate the text so it it easily readable on the plot. Maybe I could expand the plot to contain the new text, or I could move the text to a different place depending on a set of conditions about what might be near the text? I'm not sure...
I think the best answer will be one I can reuse for other projects, robust to points anywhere on the plot, and relatively easy to implement (least amount of custom functions or "hacks" that I would have to recreate for every project). Thanks a ton in advance!

Print a line/multiline in a new layer over a map using Folium

I have tried Python folium library with impressive results, but there is one feature I am missing, or in any case I can't find: I want to print a multiline in a new layer over the map.
If I check de documentation, I can only find how to add markers and poligon markers. But about printing in a new layer, I can only find examples like this one.
I need something much simplier than that. I guess I could insert a GeoJSON with the multiline info in a similar way, but I haven't been able to even find which format should that GeoJSON have.
Any idea for getting my multiline?
PD: If you don't know how to achieve this using Python/Folium, I will be happy to hear what should I add to the Javascript output to get the multiline using Leaflet (that's what Folium library is using).

Some of the functions in the earlier example are now deprecated; apparently, the preferred method is now something like:
import folium
# Coordinates are 10 points on the great circle from Boston to
# San Francisco.
# Reference: http://williams.best.vwh.net/avform.htm#Intermediate
coordinates = [
[42.3581, -71.0636],
[42.82995815, -74.78991444],
[43.17929819, -78.56603306],
[43.40320216, -82.37774519],
[43.49975489, -86.20965845],
[41.4338549, -108.74485069],
[40.67471747, -112.29609954],
[39.8093434, -115.76190821],
[38.84352776, -119.13665678],
[37.7833, -122.4167]]
# Create the map and add the line
m = folium.Map(location=[41.9, -97.3], zoom_start=4)
my_PolyLine=folium.PolyLine(locations=coordinates,weight=5)
m.add_child(my_PolyLine)
# m.save('line_example_newer.html')

I finally found a way implemented in Folium in January 2014 and not documented. Its the line method.
Here appears an example provided by the author of this addon.

Neither of the above worked for me for adding lines as a new layer to a folium.Map object (using folium 0.11). What works for me is using folium.FeatureGroup:
coords = [[[42.3554025, -71.0728116], [42.3554142, -71.0728438]],
[[42.3554142, -71.0728438], [42.3554296, -71.0728738]]]
test_map = folium.Map([42.3554025, -71.0728116], tiles='Cartodb Positron', zoom_start=15)
fg = folium.FeatureGroup("Lines")
folium.PolyLine(coords).add_to(fg)
f.add_to(test_map)
folium.LayerControl(position='bottomright').add_to(test_map)
test_map
This prints a map that has a "Lines" layer which, when toggled, will show the lines plotted at the coordinates above.