Develop Reference

How do I get a word through Selenium?

I'd like to extract and use the red alphabet of the code below through 'Selenium', so please give me some advice on how to do it
The alphabet changes randomly on every try
<td>
<input type="text" name="WKey" id="As_wkey" value="" maxlength="10" class="inputType" style="width: 300px;" title="password" />
<span id="myspam" style="padding: 2px;">
<span style="font-size: 12pt; font-weight: bold; color: red;">H</span>123
<span style="font-size: 12pt; font-weight: bold; color: red;">R</span>
<span style="font-size: 12pt; font-weight: bold; color: red;">8</span>6789
</span>
(type red word.)
</td>
here is my code
red_characters_element = driver.find_element(By.ID, 'myspam')
red_characters_elements = red_characters_element.find_elements(by = By.CSS_SELECTOR, value="span[style='font-size: 12pt; font-weight: bold; color: red;']")
print(red_characters_elements)
result []

Given all the Red colored alphabets are inside of the <span> tag. You can retrieve it using tag.
red_characters_element = driver.find_element(By.ID, 'myspam')
red_characters_elements = red_characters_element.find_elements(By.TAG_NAME, 'span')
for red_character in red_characters_elements:
print(red_character.text)
Results :
H
R
8

If you need only red letters, you can try using the java script inside the selenium:
driver.execute_script('return document.querySelectorAll("[style*=red]")')
You get an array of objects where the style has the attribute "red", with the for loop you can get the values or anything else

How do I edit attritubes of HTML tags using Beautiful Soup

I have an HTML tag like the following:
print(tag)
<td style="background-color: #e5e5e5;">
<p style="
margin: 0;
font-size: 12px;
line-height: 16px;
font-family: Arial, sans-serif;
text-align: center;
">10</p>
</td>
I can update the value from 10 to 15 in the tag with BeautifulSoup:
tag.p.contents[0].replaceWith(str(15))
However, I haven't figured out a way to update the values in the style tags; because they seem to be a part of their parent, or 'base' tags.
For example, how would I update the tag to the following?
print(tag) -->
<td style="background-color: #762157;">
<p style="
margin: 0;
font-size: 12px;
line-height: 17px;
font-family: Arial, sans-serif;
text-align: center;
">10</p>
</td>
I change the background-color to #762157 and line-height to 17px;

In the context of the html you posted, style is not a tag, but an attribute of the p tag. Here is one way to modify that p attribute (and you can apply the same for td style attribute):
from bs4 import BeautifulSoup as bs
html = '''
<td style="background-color: #e5e5e5;">
<p style="
margin: 0;
font-size: 12px;
line-height: 16px;
font-family: Arial, sans-serif;
text-align: center;
">10</p>
</td>
'''
soup = bs(html, 'html.parser')
print('OLD SOUP')
print(soup.prettify())
print('______________')
p_style_attribute = soup.select_one('p').get('style')
new_p_style_attr = '''
margin: 3;
font-size: 17px;
line-height: 26px;
font-family: Arial, sans-serif;
text-align: center;
'''
soup.select_one('p')['style'] = new_p_style_attr
print('NEW SOUP')
print(soup.prettify())
This will print out in terminal:
OLD SOUP
<td style="background-color: #e5e5e5;">
<p style="
margin: 0;
font-size: 12px;
line-height: 16px;
font-family: Arial, sans-serif;
text-align: center;
">
10
</p>
</td>
______________
NEW SOUP
<td style="background-color: #e5e5e5;">
<p style="
margin: 3;
font-size: 17px;
line-height: 26px;
font-family: Arial, sans-serif;
text-align: center;
">
10
</p>
</td>
Here is the documentation for BeautifulSoup:
https://beautiful-soup-4.readthedocs.io/en/latest/index.html

A regex approach. Collect the new information about the style into a dictionary, (tag_name, attr_value)-pairs and pass it to update_style function for an in-place modification of the soup.
The substitution is invariant wrt white spaces and ;. A **subs could also be possible.
from bs4 import BeautifulSoup as bs
import re
def update_style(soup, subs):
for tag_name, attr in subs.items():
tag = soup.find(tag_name)
k, v = attr.split(':')
tag['style'] = re.sub(rf'{k}:(.+);', f'{k}: {v.strip(" ;")};', tag['style'])
html = '''
<td style="background-color: #e5e5e5;">
<p style="
margin: 0;
font-size: 12px;
line-height: 16px;
font-family: Arial, sans-serif;
text-align: center;
">10</p>
</td>
'''
soup = bs(html, 'lxml')
subs = {'td': 'background-color: #762157;', 'p': 'line-height: 17px;'}
update_style(soup, subs)
print(soup.prettify())

Regular expression to match field name value pairs from html

I'm trying to parse an HTML email from python code to extract various details and would appreciate a regular expression or two to help achieve this as it is too complex for my limited regex understanding. e.g. look for 'Travel Date' and extract 'October 30 2018 (Tue)'.
In all cases there is a field name contained within <td> tags followed by the field value contained within another set of <td> tags. Sometimes the name and value are contained within the same row <tr> tags (Case 1) and other times they are in separate row tags (Case 2). Other items like <span> and <img> need to be skipped over as well.
Case 1
<tr>
<td colspan="2"> </td></tr>
<tr><td style="vertical-align: top; font-size: 13px; font-family: Arial; color: #777777;">Travel Date</td>
<td style="vertical-align: top; font-size: 13px; font-family: Arial; color: #444444;">October 30 2018 (Tue)</td>
</tr>
Case 2
<tr><td style="vertical-align: top;">
<span style="font-size: 10px; font-family: Arial; color: #999999; font-weight: bold; line-height: 19px; text-transform: uppercase;">Drop-off to Address</span>
</td></tr>
<tr><td style="vertical-align: top;">
<span style="font-size: 13px; font-family: Arial; color: #444444;"><img style="vertical-align:text-bottom;" src="https://d1lk4k9zl9klra.cloudfront.net/Email/Common/address_icon.png" alt="" width="14" height="14" /> 200 George St, Sydney NSW 2000, Australia</span>
</td></tr>

Instead of using regex, I would use Beautiful Soup. It makes it easier to go through HTML elements and scrape what you need. If you know the relationship between the key and value, then you could use that to extract information. Here's an example for case 1:
In [8]: from bs4 import BeautifulSoup
In [9]: text = """
...: <tr>
...: <td colspan="2"> </td></tr>
...: <tr><td style="vertical-align: top; font-size: 13px; font-family: Arial; color:
#777777;">Travel Date</td>
...: <td style="vertical-align: top; font-size: 13px; font-family: Arial; color:
#444444;">October 30 2018 (Tue)</td>
...: </tr>"""
In [11]: soup = BeautifulSoup(text, 'lxml')
In [13]: soup.find_all('td')
Out[13]:
[<td colspan="2"> </td>,
<td style="vertical-align: top; font-size: 13px; font-family: Arial; color:
#777777;">Travel Date</td>,
<td style="vertical-align: top; font-size: 13px; font-family: Arial; color:
#444444;">October 30 2018 (Tue)</td>]
In [15]: for tag in soup.find_all('td'):
...: if tag.text == "Travel Date":
...: print tag.find_next().text
...:
October 30 2018 (Tue)
Beautiful Soup gives a lot of flexibility when scraping HTML from the web.

String variable as href in lxml.builder

I am building HTML table from the list through lxml.builder and striving to make a link in one of the table cells
List is generated in a following way:
with open('some_file.html', 'r') as f:
table = etree.parse(f)
p_list = list()
rows = table.iter('div')
p_list.append([c.text for c in rows])
rows = table.xpath("body/table")[0].findall("tr")
for row in rows[2:]:
p_list.append([c.text for c in row.getchildren()])
HTML file which I parse is the same that is generated further by lxml, i.e. I set up some sort of recursion for testing purposes.
And here is how I build table
from lxml.builder import E
page = (
E.html(
E.head(
E.title("title")
),
E.body(
....
*[E.tr(
*[
E.td(E.a(E.img(src=str(col)))) if ind == 8 else
E.td(E.a(str(col), href=str(col))) if ind == 9 else
E.td(str(col)) for ind, col in enumerate(row)
]
) for row in p_list ]
When I specify link via literals all is going fine.
E.td(E.a("link", href="url_address"))
However, when I try to output list element value (which is https://blahblahblah.com) as a link
E.td(E.a(str(col), href=str(col)))
cell is empty, just nothing is showed in the cell.
If I specify link text as a literal and put str (col) into href, the link is showed normally, but instead of real href it contains the name of the generated html file.
If I output just that col value as a string
E.td(str(col))
it is showed normally, i.e. it is not empty. What is wrong with E.a and E.img elements?
Just noticed that this happens only if I build list from html file. When I build list manually, like this, all is output fine.
p_list = []
p_element = ['id']
p_element.append('value')
p_element.append('value2')
p_list.append(p_element)
Current output (pay attention to <a> and <href> tags)
<html>
<head>
<title>page</title>
</head>
<body>
<style type="text/css">
th {
background-color: DeepSkyBlue;
text-align: center;
vertical-align: bottom;
height: 150px;
padding-bottom: 3px;
padding-left: 5px;
padding-right: 5px;
}
.vertical {
text-align: center;
vertical-align: middle;
width: 20px;
margin: 0px;
padding: 0px;
padding-left: 3px;
padding-right: 3px;
padding-top: 10px;
white-space: nowrap;
-webkit-transform: rotate(-90deg);
-moz-transform: rotate(-90deg);
}</style>
<h1>title</h1>
<p>This is another paragraph, with a</p>
<table border="2">
<tr>
<th>
<div class="vertical">ID</div>
</th>
...
<th>
<div class="vertical">I blacklisted him</div>
</th>
</tr>
<tr>
<td>1020</td>
<td>ТаисияСтрахолет</td>
<td>No</td>
<td>Female</td>
<td>None</td>
<td>Санкт-Петербург</td>
<td>Росiя</td>
<td>None</td>
<td>
<a>
<img src="
"/>
</a>
</td>
<td>
<a href="
">
</a>
</td>
...
</tr>
</table>
</body>
</html>
Desired output
<html>
<head>
<title>page</title>
</head>
<body>
<style type="text/css">
th {
background-color: DeepSkyBlue;
text-align: center;
vertical-align: bottom;
height: 150px;
padding-bottom: 3px;
padding-left: 5px;
padding-right: 5px;
}
.vertical {
text-align: center;
vertical-align: middle;
width: 20px;
margin: 0px;
padding: 0px;
padding-left: 3px;
padding-right: 3px;
padding-top: 10px;
white-space: nowrap;
-webkit-transform: rotate(-90deg);
-moz-transform: rotate(-90deg);
}</style>
<h1>title</h1>
<p>This is another paragraph, with a</p>
<table border="2">
<tr>
<th>
<div class="vertical">ID</div>
</th>
...
<th>
<div class="vertical">I blacklisted him</div>
</th>
</tr>
<tr>
<td>1019</td>
<td>МихаилПавлов</td>
<td>No</td>
<td>Male</td>
<td>None</td>
<td>Санкт-Петербург</td>
<td>Росiя</td>
<td>C.-Петербург</td>
<td>
<a>
<img src="http://i.imgur.com/rejChZW.jpg"/>
</a>
</td>
<td>
link
</td>
...
</tr>
</table>
</body>
</html>

Got it myself. The problem was not in generating but in parsing HTML. Parsing function didn't fetch IMG and A tags nested in TD and these elements of the list were empty. Due to the rigorous logic of the program (fetching from file + fetching from site API) I wasn't able to detect the cause of the issue.
The correct parsing logic should be:
for row in rows[1:]:
data.append([
c.find("a").text if c.find("a") is not None else
c.find("img").attrib['src'] if c.find("img") is not None else
c.text
for c in row.getchildren()
])

Just a guess but it looks like 'str()' could be escaping it. Try E.td(E.a(col, href=col))

Handling unicode in Python 2.7

I'm getting this error:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2661' in position 1409: ordinal not in range(128)
I'm very green to programming still, so have mercy on me and my ignorance. But I understand the error to be that it's not able to handle unicode characters. There's that at least one unicode char, but there could be countless others that'll perk up in that feed.
I've done some looking for others who've had similar problems, but I can't can't find a solution I understand or can make work.
#import library to do http requests:
import urllib
from xml.dom.minidom import parseString, parse
f = open('games.html', 'w')
document = urllib.urlopen('https://itunes.apple.com/us/rss/topfreemacapps/limit=300/genre=12006/xml')
dom = parse(document)
image = dom.getElementsByTagName('im:image')
title = dom.getElementsByTagName('title')
price = dom.getElementsByTagName('im:price')
address = dom.getElementsByTagName('id')
imglist = []
titlist = []
pricelist = []
addlist = []
i = 0
j = 20
k = 40
f.write('''\
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
<!--
A:link {text-decoration: none; color: #246DA8;}
A:visited {text-decoration: none; color: #246DA8;}
A:active {text-decoration: none; color: #40A9E3;}
A:hover {text-decoration: none; color: #40A9E3;}
.box {
vertical-align:middle;
width: 180px;
height: 120px;
border: 1px solid #99c;
padding: 5px;
margin: 0px;
margin-left: auto;
margin-right: auto;
-moz-border-radius: 5px;
border-radius: 5px;
-webkit-border-radius: 5px;
background-color:#ffffff;
font-family: Arial, Helvetica, sans-serif; color: black;
font-size: small;
font-weight: bold;
}
-->
</style>
</head>
<body>
''')
for i in range(0,len(image)):
if image[i].getAttribute('height') == '53':
imglist.append(image[i].firstChild.nodeValue)
for i in range(1,len(title)):
titlist.append(title[i].firstChild.nodeValue)
for i in range(0,len(price)):
pricelist.append(price[i].firstChild.nodeValue)
for i in range(1,len(address)):
addlist.append(address[i].firstChild.nodeValue)
for i in range(0,20):
f.write('''
<div style="width: 600px;">
<div style="float: left; width: 200px;">
<div class="box" align="center">
<div align="center">
''' + titlist[i] + '''<br>
<img src="''' + imglist[i] + '''" alt="" width="53" height="53" border="0" ><br>
<span>''' + pricelist[i] + '''</span>
</div>
</div>
</div>
<div style="float: left; width: 200px;">
<div class="box" align="center">
<div align="center">
''' + titlist[i+j] + '''<br>
<img src="''' + imglist[i+j] + '''" alt="" width="53" height="53" border="0" ><br>
<span>''' + pricelist[i+j] + '''</span>
</div>
</div>
</div>
<div style="float: left; width: 200px;">
<div class="box" align="center">
<div align="center">
''' + titlist[i+k] + '''<br>
<img src="''' + imglist[i+k] + '''" alt="" width="53" height="53" border="0" ><br>
<span>''' + pricelist[i+k] + '''</span>
</div>
</div>
</div>
<br style="clear: left;" />
</div>
<br>
''')
f.write('''</body>''')
f.close()

The basic problem is that you're concatenating the Unicode strings with ordinary byte-strings without converting them using a proper encoding; in these cases, ASCII is used by default (which, clearly, can't handle extended characters).
The line in your script that does this is too long to quote, but another practical example which displays the same problem could look like this:
parameter = u"foo \u2661"
sys.stdout.write(parameter + " bar\n")
You will need to instead encode the Unicode strings with an explicitly specified encoding, e.g. like this:
parameter = u"foo \u2661"
sys.stdout.write(parameter.encode("utf8") + " bar\n")
In your case, you can do this in your loops so as to not have to specify it on every concatenation:
for i in range(1,len(title)):
titlist.append(title[i].firstChild.nodeValue.encode("utf8"))
--
Also, while we're at it, you can improve your code by not iterating through the elements using an integer index. For instance, instead of this:
title = dom.getElementsByTagName('title')
for i in range(1,len(title)):
titlist.append(title[i].firstChild.nodeValue.encode("utf8"))
... you can do this instead:
for title in dom.getElementsByTagName('title')
titlist.append(title.firstChild.nodeValue.encode("utf8"))