I am trying to add a vhost entry to tomcat server.xml using python lxml
import io
from lxml import etree
newdoc = etree.fromstring('<Host name="getrailo.com" appBase="webapps"><Context path="" docBase="/var/sites/getrailo.org" /><Alias>www.getrailo.org</Alias><Alias>my.getrailo.org</Alias></Host>')
doc = etree.parse('/root/server.xml')
root = doc.getroot()
for node1 in root.iter('Service'):
for node2 in node1.iter('Engine'):
node2.append(newdoc)
doc.write('/root/server.xml')
The problem is that it is removing the
<?xml version='1.0' encoding='utf-8'?>
line on top of the file from the output and the vhost entry is all in one line .How can I add the xml element in a pretty way like
<Host name="getrailo.org" appBase="webapps">
<Context path="" docBase="/var/sites/getrailo.org" />
<Alias>www.getrailo.org</Alias>
<Alias>my.getrailo.org</Alias>
</Host>
First you need to parse existing file with remove_blank_text so that it's clean and with no extra spaces that I think is a problem in this case
parser = etree.XMLParser(remove_blank_text=True)
newdoc = etree.fromstring('/root/server.xml' parser=parser)
Then you're safe to write it back to disk with pretty_print and xml_declaration set in doc.write()
doc.write('/root/server.xml',
xml_declaration=True,
encoding='utf-8',
pretty_print=True)
Related
I'm trying to extract URLs from a sitemap like this: https://www.bestbuy.com/sitemap_c_0.xml.gz
I've unzipped and saved the .xml.gz file as an .xml file. The structure looks like this:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xhtml="http://www.w3.org/1999/xhtml" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
<url>
<loc>https://www.bestbuy.com/</loc>
<priority>0.0</priority>
</url>
<url>
<loc>https://www.bestbuy.com/site/3d-printers/3d-printer-filament/pcmcat335400050008.c?id=pcmcat335400050008</loc>
<priority>0.0</priority>
</url>
<url>
<loc>https://www.bestbuy.com/site/3d-printers/3d-printing-accessories/pcmcat748300527647.c?id=pcmcat748300527647</loc>
<priority>0.0</priority>
</url>
I'm attempting to use ElementTree to extract all of the URLs within the loc nodes throughout this file, but struggling to get it working right.
Per the documentation, I'm trying something like this:
import xml.etree.ElementTree as ET
tree = ET.parse('my_local_filepath')
root = tree.getroot()
value = root.findall(".//loc")
However, nothing gets loaded into value. My goal is to extract all of the URLs between the loc nodes and print it out into a new flat file. Where am I going wrong?
You were close in your attempt but like mzjn said in a comment, you didn't account for the default namespace (xmlns="http://www.sitemaps.org/schemas/sitemap/0.9").
Here's an example of how to account for the namespace:
import xml.etree.ElementTree as ET
tree = ET.parse('my_local_filepath')
ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
for elem in tree.findall(".//sm:loc", ns):
print(elem.text)
output:
https://www.bestbuy.com/
https://www.bestbuy.com/site/3d-printers/3d-printer-filament/pcmcat335400050008.c?id=pcmcat335400050008
https://www.bestbuy.com/site/3d-printers/3d-printing-accessories/pcmcat748300527647.c?id=pcmcat748300527647
Note that I used the namespace prefix sm, but you could use any NCName.
See here for more information on parsing XML with namespaces in ElementTree.
We can iterate through the URLs, toss them into a list and write them to a file as such:
from xml.etree import ElementTree as ET
tree = ET.parse('test.xml')
root = tree.getroot()
name_space = '{http://www.sitemaps.org/schemas/sitemap/0.9}'
urls = []
for child in root.iter():
for block in child.findall('{}url'.format(name_space)):
for url in block.findall('{}loc'.format(name_space)):
urls.append('{}\n'.format(url.text))
with open('sample_urls.txt', 'w+') as f:
f.writelines(urls)
note we need to append the name space from the open urlset definition to properly parse the xml
I have a xml file as follows
<Person>
<name>
My Name
</name>
<Address>My Address</Address>
</Person>
The tag has extra new lines, Is there any quick Pythonic way to trim this and generate a new xml.
I found this but it trims only which are between tags not the value
https://skyl.org/log/post/skyl/2010/04/remove-insignificant-whitespace-from-xml-string-with-python/
Update 1 - Handle following xml which has tail spaces in <name> tag
<Person>
<name>
My Name<shortname>My</short>
</name>
<Address>My Address</Address>
</Person>
Accepted answer handle above both kind of xml's
Update 2 - I have posted my version in answer below, I am using it to remove all kind of whitespaces and generate pretty xml in file with xml encodings
https://stackoverflow.com/a/19396130/973699
With lxml you can iterate over all elements and check if it has text to strip():
from lxml import etree
tree = etree.parse('xmlfile')
root = tree.getroot()
for elem in root.iter('*'):
if elem.text is not None:
elem.text = elem.text.strip()
print(etree.tostring(root))
It yields:
<Person><name>My Name</name>
<Address>My Address</Address>
</Person>
UPDATE to strip tail text too:
from lxml import etree
tree = etree.parse('xmlfile')
root = tree.getroot()
for elem in root.iter('*'):
if elem.text is not None:
elem.text = elem.text.strip()
if elem.tail is not None:
elem.tail = elem.tail.strip()
print(etree.tostring(root, encoding="utf-8", xml_declaration=True))
Accepted answer given by Birei using lxml does the job perfectly, but I wanted to trim all kind of white/blank space, blank lines and regenerate pretty xml in a xml file.
Following code did what I wanted
from lxml import etree
#discard strings which are entirely white spaces
myparser = etree.XMLParser(remove_blank_text=True)
root = etree.parse('xmlfile',myparser)
#from Birei's answer
for elem in root.iter('*'):
if elem.text is not None:
elem.text = elem.text.strip()
if elem.tail is not None:
elem.tail = elem.tail.strip()
#write the xml file with pretty print and xml encoding
root.write('xmlfile', pretty_print=True, encoding="utf-8", xml_declaration=True)
You have to do xml parsing for this one way or another, so maybe use xml.sax and copy to the output stream at each event (skipping ignorableWhitespace), and add tag markers as needed. Check the sample code here http://www.knowthytools.com/2010/03/sax-parsing-with-python.html.
You can use beautifulsoup. Do traverse all elements and for each one that contains some text, replace it with its stripped version:
from bs4 import BeautifulSoup
soup = BeautifulSoup(open('xmlfile', 'r'), 'xml')
for elem in soup.find_all():
if elem.string is not None:
elem.string = elem.string.strip()
print(soup)
Assuming xmlfile with the content provided in the question, it yields:
<?xml version="1.0" encoding="utf-8"?>
<Person>
<name>My Name</name>
<Address>My Address</Address>
</Person>
I'm working with an older version of Python (2.3), and I'm currently stuck with the standard library. To show an answer that's greatly backwards compatible, I've written this with xml.dom and xml.minidom functions.
import codecs
from xml.dom import minidom
# Read in the file to a DOM data structure.
original_document = minidom.parse("original_document.xml")
# Open a UTF-8 encoded file, because it's fairly standard for XML.
stripped_file = codecs.open("stripped_document.xml", "w", encoding="utf8")
# Tell minidom to format the child text nodes without any extra whitespace.
original_document.writexml(stripped_file, indent="", addindent="", newl="")
stripped_file.close()
While it's not BeautifulSoup, this solution is pretty elegant and uses the full force of the lower-level API. Note that the actual formatting is just one line :)
Documentation of API calls used here:
minidom.parse
minidom.Node.writexml
codecs.open
i am very new to programming and python. I am trying to find and replace a text in an xml file. Here is my xml file
<?xml version="1.0" encoding="UTF-8"?>
<!--Arbortext, Inc., 1988-2008, v.4002-->
<!DOCTYPE doc PUBLIC "-//MYCOMPANY//DTD XSEIF 1/FAD 110 05 R5//EN"
"XSEIF_R5.dtd">
<doc version="XSEIF R5"
xmlns="urn:x-mycompany:r2:reg-doc:1551-fad.110.05:en:*">
<meta-data></meta-data>
<front></front>
<body>
<chl1><title xml:id="id_881i">Installation</title>
<p>To install SDK, perform the tasks mentioned in the following
table.</p>
<p><input>ln -s /sim/<var>user_id</var>/.VirtualBox $home/.VirtualBox</input
></p>
</chl1>
</body>
</doc>
<?Pub *0000021917 0?>
I need to replace all entries of "virtual box" with "Xen". For this i tried Elementtree. But i dont know how to replace and write back to the file. Here is my try.
import xml.etree.ElementTree as ET
tree=ET.parse('C:/My_location/1_1531-CRA 119 1364_2.xml')
doc=tree.getroot()
iterator=doc.getiterator()
for body in iterator:
old_text=body.replace("Virtualbox", "Xen")
The texts are available in many sub tags under body.I got the method to remove the subelement and append a new element, but didnt get to replace only the texts.
Replace text, tail attributes.
import lxml.etree as ET
with open('1.xml', 'rb+') as f:
tree = ET.parse(f)
root = tree.getroot()
for elem in root.getiterator():
if elem.text:
elem.text = elem.text.replace('VirtualBox', 'Xen')
if elem.tail:
elem.tail = elem.tail.replace('VirtualBox', 'Xen')
f.seek(0)
f.write(ET.tostring(tree, encoding='UTF-8', xml_declaration=True))
f.truncate()
Probably the simplest way is to do:
ifile = open('input_file','r')
ofile = open('output_file','w')
for line in ifile.readlines():
ofile.write(line.replace('VirtualBox','Xen'))
ifile.close()
ofile.close()
I have a XML file that starts like this:
<?xml version="1.0" encoding="utf-8"?>
<Recipe xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
I need to read it in, modify it, then write it back out. Here is a code snippet:
from xml.etree import ElementTree
with open('base.xml', 'rt') as f:
tree = ElementTree.parse(f)
recipe = tree.find('')
t = recipe.find('Targets_Params/Target_Table/Target_Name')
t.text = "new Value"
output_file = open('new.xml', 'w' )
output_file.write(ElementTree.tostring(recipe))
output_file.close()
My problem is that when I write the file out I do not get the first line at all, and the second line comes out with just:
<Recipe>
How I can read in the file, modify it, and write it out while preserving the original structure?
I am learning ElementTree in python. Everything seems fine except when I try to parse the xml file with prefix:
test.xml:
<?xml version="1.0"?>
<abc:data>
<abc:country name="Liechtenstein" rank="1" year="2008">
</abc:country>
<abc:country name="Singapore" rank="4" year="2011">
</abc:country>
<abc:country name="Panama" rank="5" year="2011">
</abc:country>
</abc:data>
When I try to parse the xml:
import xml.etree.ElementTree as ET
tree = ET.parse('test.xml')
I got the following error:
xml.etree.ElementTree.ParseError: unbound prefix: line 2, column 0
Do I need to specify something in order to parse a xml file with prefix?
Add the abc namespace to your xml file.
<?xml version="1.0"?>
<abc:data xmlns:abc="your namespace">
I encountered the same issue while processing xml file. You can use below code before parse your XML file. This will resolve your issue.
parser1 = etree.XMLParser(encoding="utf-8", recover=True)
tree1 = ElementTree.parse('filename.xml', parser1)
See if this works:
from bs4 import BeautifulSoup
xml_file = "test.xml"
with open(xml_file, "r", encoding="utf8") as f:
contents = f.read()
soup = BeautifulSoup(contents, "xml")
items = soup.find_all("country")
print (items)
The above will produce an array which you can then manipulate to achieve your aim (e.g. remove html tags etc.):
[<country name="Liechtenstein" rank="1" year="2008">
</country>, <country name="Singapore" rank="4" year="2011">
</country>, <country name="Panama" rank="5" year="2011">
</country>]