Develop Reference

Python Won't Enter Try/Except In For Loop On 50th Iteration

First off I want to apologize to everyone who's about to read this code... I know it's a mess.
For anyone who is able to decipher it: I have a list of ~16,500 website URL's that I am scraping and then using googles NLP to categorize. The list of URL's is created with the following chunk of code, as far as I can tell nothing is broken here.
url_list = open("/Users/my_name/Documents/Website Categorization and Scrapper Python/url_list copy", "r")
indexed_url_list = url_list.readlines()
clean_url_list = []
clean_url_list = [x[:-1] for x in indexed_url_list]
When I print the length of this list it correctly gives me the count of ~16,500.
The main block of code is as follows:
for x in clean_url_list:
print('1')
url = x
print('1.1')
try:
r = scraper.get(url, headers = headers,)
print('1.2')
soup = BeautifulSoup(r.text, 'html.parser')
print('1.3')
title = soup.find('title').text
print('1.4')
description = soup.find('meta', attrs={'name': 'description'})["content"]
print('2')
if "content" in str(description):
description = description.get("content")
else:
description = ""
h1 = soup.find_all('h1')
h1_all = ""
for x in range (len(h1)):
if x == len(h1) -1:
h1_all = h1_all + h1[x].text
else:
h1_all = h1_all + h1[x].text + ". "
paragraphs_all = ""
paragraphs = soup.find_all('p')
for x in range (len(paragraphs)):
if x == len(paragraphs) -1:
paragraphs_all = paragraphs_all + paragraphs[x].text
else:
paragraphs_all = paragraphs_all + paragraphs[x].text + ". "
h2 = soup.find_all('h2')
h2_all = ""
for x in range (len(h2)):
if x == len(h2) -1:
h2_all = h2_all + h2[x].text
else:
h2_all = h2_all + h2[x].text + ". "
h3 = soup.find_all('h3')
h3_all = ""
for x in range (len(h3)):
if x == len(h3) -1:
h3_all = h3_all + h3[x].text
else:
h3_all = h3_all + h3[x].text + ". "
allthecontent = ""
allthecontent = str(title) + " " + str(description) + " " + str(h1_all) + " " + str(h2_all) + " " + str(h3_all) + " " + str(paragraphs_all)
allthecontent = str(allthecontent)[0:999]
print(allthecontent)
except Exception as e:
print(e)
When I run this it successfully categorizes the first 49 URL's, but ALWAYS stops on the 50th, no matter what URL it is. No error is thrown, and even if it did the try/except should handle it. Using the print statements to debug it seems to not enter the "try" section on the 50th iteration for whatever reason and it's always the 50th iteration
Any help would be much appreciated and I hope you have some good eye wash to wipe away the code you just had to endure.

I helped look at this at work. The actual issue was a bad 50th url that would not return. Adding a timeout allowed the code to escape the try/catch block and move on to the next url in a manageable fashion.
try:
r = scraper.get(url, headers = headers, timeout=5)
except:
continue # handle next url in list

Refresh multiple print lines within terminal

Super simple not complex and don't need it to be. Just wanted to add the current prices of crypto and my equity of these prices as well as a total to display on screen. I want these values to refresh in place so they don't create new lines every time it refreshes.
Here is what I have:
import requests
import math
from datetime import datetime
URL = "https://min-api.cryptocompare.com/data/price?fsym={}&tsyms={}"
def get_price(coin, currency):
try:
response = requests.get(URL.format(coin, currency)).json()
return response
except:
return False
while True:
date_time = datetime.now()
date_time = date_time.strftime("%m/%d/%Y %H:%M:%S")
currentPrice = get_price("ETH", "USD")
totalPrice = currentPrice["USD"]*2
currentPrice0 = get_price("DOGE", "USD")
totalPrice0 = currentPrice0["USD"]*100
currentPrice1 = get_price("ADA", "USD")
totalPrice1 = currentPrice1["USD"]*20
currentPrice2 = get_price("SOL", "USD")
totalPrice2 = currentPrice2["USD"]*4
currentPrice3 = (totalPrice + totalPrice0 + totalPrice1 + totalPrice2)
print("ETH | PRICE: $", currentPrice["USD"], " EQUITY: $", totalPrice)
print("DOGE | PRICE: $", currentPrice0["USD"], " EQUITY: $", totalPrice0)
print("ADA | PRICE: $", currentPrice1["USD"], " EQUITY: $", totalPrice1)
print("SOL | PRICE: $", currentPrice2["USD"], " EQUITY: $", totalPrice2)
print(" ")
print("TOTAL: ", currentPrice3)

Lab: calculate discounts

I'm doing an exercise and so far so good as the code (after some help from other threads) now works almost fine, but...can't get the right results as a math point of view.
Here it's the code:
#getting base prices from user
item1 = float(input('Enter the price of the first item: '))
item2 = float(input('Enter the price of the second item: '))
clubc = raw_input('Does customer have a club card? (Y/N): ')
tax = float(input('Enter tax rate, e.g. 5.5 for 5.5% tax: '))
basep = (item1 + item2)
print('Base price = ', basep)
#setting variables for calculation
addtax = (1 + (tax / 100))
#conditions for output
if item1 >= item2 and clubc == 'N':
priceafterd = float(item1 + (item2 / 2))
print('Price after discounts = ', priceafterd)
totalprice = (priceafterd * addtax)
print('Total price = ', totalprice)
elif item2 >= item1 and clubc == 'N':
priceafterd = float(item2 + (item1 / 2))
print('Price after discounts = ', priceafterd)
totalprice = (priceafterd * addtax)
print('Total price = ', totalprice)
if item1 >= item2 and clubc == 'Y':
priceafterd = float((item1 + (item2 / 2)) * 0.9)
print('Price after discounts = ', priceafterd)
totalprice = (priceafterd * var3)
print('Total price = ' + totalprice)
else:
priceafterd = float((item2 + (item1 / 2)) * 0.9)
print('Price after discounts = ', priceafterd)
totalprice = (priceafterd * var3)
print('Total price = ' + totalprice)
The exercise requires to write a program that computes how much a customer has to pay after purchasing two items, depending on a promo, club card and taxes.
The problem is with the results. As an example of inputs:
Enter price of the first item: 10
Enter price of the second item: 20
Does customer have a club card? (Y/N): y
Enter tax rate, e.g. 5.5 for 5.5% tax: 8.25
Base price = 30.00
Price after discounts = 22.50
Total price = 24.36
Instead, I got:
line 33, in <module>
print('Total price = ' + totalprice)
TypeError: cannot concatenate 'str' and 'float' objects
What's wrong with the syntax? Thanks a lot!

The problem
In the second conditional you wrote print('Total price = ' + totalprice) line instead of the print('Total price = ', totalprice), and the problem is in that:
totalprice has float type, meanwhile 'Total price = ' is the str and what you are trying to do is almost like str() + float(), and because python doesn't know how to concatenate string and float number it raises an exception.
How to solve
1) Use the same print('Total price = ', totalprice) line everywhere
Why does it work and print('Total price = ' + totalprice) does not?
Because print automatically converts everything to string representation, you can imagine print('Total price = ', totalprice) expression like that:
print(str('Total price = ') + " " + str(totalprice))
2) Convert float to str and concatenate strs
print('Total price = ' + str(totalprice))
str(totalprice) converts totalprice from float to the str and python knows how to concatenate strings together.
3) Formatting
"Total price = {}".format(3.14)" is equivalent to the "Total price = 3.14" string,
so print("Total price = {}".format(totalprice)) also will work
in python 3 we also have f-stings:
f"Total price = {3.14}" == "Total price = 3.14"
print(f"Total price = {totalprice}")

How to store a value and change it when it is updated in python?

So I'm trying to create a bitcoin price tracker and with it, it will tell you whether the price went up or down. For example, when you start it, it shows the price and an increase of $0.00, then when it increases or decreases it shows that. The problem is after it tests again, it reverts back to 0, not staying at the increase or decrease amount.
Heres what I have tried and everything works but when it changes, it only shows it until it tests for a change, after one second.
###############
import requests
import time
import os
#############
#######
bct = 0.0
bctChange = 0
errorLevel = 0
################
os.system('cls')
################
#print('The current price of Bitcoin is $10,000.00')
#print('Connected: True')
#print('Increase of $2.50') #use abs() for absolute value
#print('ErrorLevel: 0')
#print('ErrorLevel is the amount of request errors\nthere have been to the bitcoin api')
########################
def place_value(number):
return ("{:,}".format(number))
#####################################################################
r = requests.get('https://api.coindesk.com/v1/bpi/currentprice.json')
bctPriceStart = place_value(round(r.json()['bpi']['USD']['rate_float'], 2))
###########################################################################
while True:
try:
#############
bctLast = bct
#####################################################################
r = requests.get('https://api.coindesk.com/v1/bpi/currentprice.json')
bct = round(r.json()['bpi']['USD']['rate_float'], 2)
####################################################
if (bctChange != bctChange):
bctChange = bctLast - bct
###########################
################
os.system('cls')
print('The current price of Bitcoin is $' + place_value(bct))
#############################################################
#################
if (bctLast > 0):
print('Increase of $' + place_value(abs(round(bctChange, 2))))
time.sleep(1)
###############
###################
elif (bctLast < 0):
print('Decrease of $' + place_value(abs(round(bctChange, 2))))
time.sleep(1)
###############
except requests.ConnectionError or requests.ConnectTimeout or requests.HTTPError or requests.NullHandler or requests.ReadTimeout or requests.RequestException or requests.RequestsDependencyWarning or requests.Timeout or requests.TooManyRedirects:
#Do error function
os.system('cls')
print('There was an error...')

If I understand what you want to do, replacing your script with this should work:
import requests
import time
bct = 0.0
bctChange = 0
def place_value(number):
return ("{:,}".format(number))
r = requests.get('https://api.coindesk.com/v1/bpi/currentprice.json')
bct = round(r.json()['bpi']['USD']['rate_float'], 2)
print('The starting price of Bitcoin is $' + place_value(bct))
while True:
try:
bctLast = bct
r = requests.get('https://api.coindesk.com/v1/bpi/currentprice.json')
bct = round(r.json()['bpi']['USD']['rate_float'], 2)
bctChange = bctLast - bct
if bctChange > 0:
print('The current price of Bitcoin is $' + place_value(bct))
print('Increase of $' + place_value(abs(round(bctChange, 2))))
elif bctChange < 0:
print('The current price of Bitcoin is $' + place_value(bct))
print('Decrease of $' + place_value(abs(round(bctChange, 2))))
time.sleep(1)
except requests.ConnectionError or requests.ConnectTimeout or requests.HTTPError or requests.NullHandler or requests.ReadTimeout or requests.RequestException or requests.RequestsDependencyWarning or requests.Timeout or requests.TooManyRedirects:
print('There was an error...')
The problem is your if...elif statements were based on btcLast not btcChange, and you don't need the if statement around bctChange = bctLast - bct. You were also calling os.system('cls') which was clearing your printed output rather than printing it persistently.

Using data from a website, match it to a table and pull data

After hours and hours of looking around and reading documents, I have to call for help and sanity. I'm no python expert or even HTML expert, so I'd appreciate every small bit of help that I can get.
I can pay for a little time if needed...
What I'm trying to do is:
Two webpages. One is in my server and one isn't. The one that isn't in my server (order.asp), has this line:
<FONT CLASS="fv">Estimated Weight of Order:</FONT></TD><TD ALIGN="RIGHT"><FONT CLASS="fv">XX.XXoz XX.XXg</FONT>
I need something that I can put in my server, queries the weight from the page that isn't on my server (order.asp page) and matches the weight with a shipping price that I would have on my page (as a table or maybe with ifs).
There will be different order pages (order1.asp order2.asp order3.asp) with different weights. The script or whatever should do that for ea. wpage.
Here's a flowchart I just made to help understand:
http://www.gliffy.com/go/publish/image/5123674/L.png
A very helpful user already gave me this piece of code:
html = open("html.txt").read()
out = open("foundWeights.txt", "w")
#split html on order number
legoOrders = html.split("Order #")
for order in legoOrders[1:]:
print order
orderNumber = order.split("<")[0]
weightString = order.split('Estimated Weight of Order:</FONT></TD><TD ALIGN="RIGHT"><FONT CLASS="fv">')[1]
splitWeightString = weightString.split(' ')
splitStringFinal = splitWeightString[1].split("<")
grams = splitStringFinal[0]
ozs = weightString.split('&nbsp')[0]
out.write(str(orderNumber) + "\t" + str(grams) + "\t" + str(ozs) + "\n"
Which pulls out the order number, and weight in ozs and grams.
What I don't know how to do is, having a table of shipping prices and weight ranges, match that weight to a shipping price, pull the shipping price and... well, do some things with it.
Ideally, if I knew how to do that (I don't know nothing in python, but I do know programming basics and C), I could pull the order total too, sum the order total and the shipping price, and FINALLY get a grand total.
I hope I made it clear.
All the best,
Gerald
This is how the page order.asp will look like (varying the weight and order total numbers):
http://pastebin.com/uH18CF5k
Getting there slowly. I added this to your code #duhaime
#get Shipping Range
test = "B"
if ( 0 < grams < 100 ):
test = "A"
if ( 100 < grams < 500):
test = "A"
Along with the str(test) in the out.write.
However, it prints a B when I run it, but it should print an A. Know what's wrong?

Thanks again for your feedback, #Brick Top. This code pulls down the Order Number, Grand Total, Order Total, Shipping Cost, Weight in Grams, and Weight in Ozs from the html, and writes them to a tab-separated text file that you can open and easily review in Excel.
EDIT: The script also calculates the shipping class, based on the weight of the order in grams. The Shipping Class is indicated in the final column of the out file:
from decimal import *
html = open("html.txt").read()
out = open("legoShipping.txt", "w")
out.write("Order Number" + "\t" +
"Grand Total (currency)" + "\t" +
"Order Total (currency)" + "\t" +
"Shipping Cost (currency)" + "\t" +
"Order Weight (grams)" + "\t" +
"Order Weight (oz.)" + "\t" +
"Shipping Class" + "\n")
#split html on order number
legoOrders = html.split("Order #")
for order in legoOrders[1:]:
orderNumber = order.split("<")[0]
#get Grand Total
grand = order.split("<TD>Grand Total:</TD>")[1].split('<TD ALIGN="RIGHT"><B>')[1].split("<")[0].split(' ')
grandCurrency = grand[0]
grandTotal = grand[1]
#get Order Total
orderTotalAndCurrency = order.split('<TD>Order Total:</TD>')[1].split('<TD ALIGN="RIGHT">')[1].split("<")[0].split(' ')
orderCurrency = orderTotalAndCurrency[0]
orderTotal = orderTotalAndCurrency[1]
#get Shipping Cost
shipping = order.split("<TD>Shipping:</TD>")[1].split('<TD ALIGN="RIGHT">')[1].split("<")[0].split(' ')
shippingCurrency = shipping[0]
shippingCost = shipping[1]
#get Weights
weightString = order.split('Estimated Weight of Order:</FONT></TD><TD ALIGN="RIGHT"><FONT CLASS="fv">')[1]
splitWeightString = weightString.split(' ')
splitStringFinal = splitWeightString[1].split("<")
grams = splitStringFinal[0]
ozs = weightString.split('&nbsp')[0]
#convert grams to mathematical value
gramsValue = Decimal(grams[:-1])
#create default shipping class value. Set it to ""
shippingClass = ""
if gramsValue > 0:
if gramsValue < 100:
shippingClass = "A"
if gramsValue >= 100:
if gramsValue < 200:
shippingClass = "B"
out.write(str(orderNumber) + "\t" +
str(grandTotal) + " (" + str(grandCurrency) + ")" + "\t" +
str(orderTotal) + " (" + str(orderCurrency) + ")" + "\t" +
str(shippingCost) + " (" + str(shippingCurrency) + ")" + "\t" +
str(grams) + "\t" +
str(ozs) + "\t" +
str(shippingClass) + "\n")
Output in OpenOffice Calc: