Develop Reference

python error while looping from a list json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm making a script that fills a text document with responses from an api. The api is being asked to convert usernames from a list to universally unique identifiers. I keep getting this error and can't find a way around it. "json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"
Sample of accounts.txt
knapplace
Coppinator
tynow
Pman59
ButterMusty
FlyHighGuy13
Seyashi
fluzzygirl1
SquidMan55
leonrules9
BarthGimble
MTR_30
Darkshadow402
Deathmyster
Team_Everlook
Sheathok
KCFrost
mendog
Allfaal117
theLP25D
Zimyx
Blurrnis
redboy678
moose_breeder
kaser12345
import requests
import json
file1 = open('accounts.txt', 'r')
usernames = []
for line in file1:
stripped_line = line.strip()
usernames.append(stripped_line)
file1.close()
for x in usernames:
username = str(x)
url = ("https://api.mojang.com/users/profiles/minecraft/"+username+"?at=1462770000")
y = requests.get(url)
y_data = y.json()
uuid = y_data['id']
uuids = []
uuids.append(uuid)
file2 = open('uuids.txt', 'w')
file2.writelines(uuids)
file2.close()
file2 = open('uuids.txt', 'r')
lines = file2.readlines()

Note: #Ali makes a great point about checking for an empty reply. With that fix it works like a champ for me with a few other minor changes:
Used usernames provided by OP instead of reading them in from a file.
Moved initialization of uuids out of for loop to avoid it being reset for each username.
Modfied file i/o stuff to what I am more used to working with. ;^)
import requests
import json
usernames = [
"knapplace",
"Coppinator",
"tynow",
]
uuids = []
for x in usernames:
username = str(x)
url = ("https://api.mojang.com/users/profiles/minecraft/"+username+"?at=1462770000")
y = requests.get(url)
if len(y.content) == 0:
continue # Skip processing this username
y_data = y.json()
uuid = y_data['id']
uuids.append(uuid)
with open('uuids.txt', 'w') as f:
for uuid in uuids:
f.write(uuid + '\n')
with open('uuids.txt', 'r') as f:
read_data = f.read()
print(read_data)
Output:
c9998bafea3146d5935f4e215b6b4351
5c321f81409847a0907c4b30c342217f
9f206def69bf407fbab6de7c9b70ff80

I checked the URL you pasted. If the user does not exist, the API does not return any content but still returns a successful status. That is what the error means — it expected there to be a JSON object starting at char 0.
Essentially, you need to handle the case when the response is empty before you try to execute a y.json() by checking y.content. If y.content is empty, skip processing the current username and go to the next one.
y = requests.get(url)
if len(y.content) == 0:
continue # Skip processing this username
# The rest of the code only runs if y.content is not empty.
y_data = y.json()
uuid = y_data['id']

Python reading URLs from file until last line

I have script which basically checks domain from the text file and finds its email. I want to add multiple domain names(line by line) then script should take each domain run the function and goes to second line after finishing. I tried to google for specific solution but not sure how do i find appropriate answer.
f = open("demo.txt", "r")
url = f.readline()
extractUrl(url)
def extractUrl(url):
try:
print("Searching emails... please wait")
count = 0
listUrl = []
req = urllib.request.Request(
url,
data=None,
headers={
'User-Agent': ua.random
})
try:
conn = urllib.request.urlopen(req, timeout=10)
status = conn.getcode()
contentType = conn.info().get_content_type()
html = conn.read().decode('utf-8')
emails = re.findall(
r '[A-Za-z0-9._%+-]+#[A-Za-z0-9.-]+\.[A-Za-z]{2,4}', html)
for email in emails:
if (email not in listUrl):
count += 1
print(str(count) + " - " + email)
listUrl.append(email)
print(str(count) + " emails were found")

Python files are iterable, so it's basically a simple as:
for line in f:
extractUrl(line)
But you may want to do it right (ensure you close the file whatever happens, ignore possible empty lines etc):
# use `with open(...)` to ensure the file will be correctly closed
with open("demo.txt", "r") as f:
# use `enumerate` to get line numbers too
#- we might need them for information
for lineno, line in enumerate(f, 1):
# make sure the line is clean (no leading / trailing whitespaces)
# and not empty:
line = line.strip()
# skip empty lines
if not line:
continue
# ok, this one _should_ match - but something could go wrong
try:
extractUrl(line)
except Exception as e:
# mentioning the line number in error report might help debugging
print("oops, failed to get urls for line {} ('{}') : {}".format(lineno, line, e))

Python File String Replace Dict and Tuple

have a Dict with multiple values in a tuple.
newhost = {'newhost.com': ('1.oldhost.com',
'2.oldhost.com',
'3.oldhost.com',
'4.oldhost.com')
}
I wanna open a existing file and search for lines in this file that contains a value of the oldhosts. A file can have multiple Account Lines. In example
Account: 1.oldhost.com username
Account: someotherhost username
When the line with 1.oldhost.com or 2.oldhost.com or 3.oldhost.com and so on is found i wanna replace it with the key form the dict newhost.com.
Can anyone help me? Searched alot, but didnt find the right thing.
Regards

Maybe something like this could get you started
infile_name = 'some_file.txt'
# Open and read the incoming file
with open(infile_name, 'r') as infile:
text = infile.read()
# Cycle through the dictionary
for newhost, oldhost_list in host_dict.items():
# Cycle through each possible old host
for oldhost in oldhost_list:
text.replace(oldhost, newhost)
outfile_name = 'some_other_file.txt'
# Write to file
with open(outfile_name, 'w') as outfile:
outfile.write(text)
Not claiming this to be the best solution, but it should be a good start for you.

To easily find the new host for a given old host, you should convert your data structure:
# your current structure
new_hosts = {
'newhost-E.com': (
'1.oldhost-E.com',
'2.oldhost-E.com',
),
'newhost-A.com': (
'1.oldhost-A.com',
'2.oldhost-A.com',
'3.oldhost-A.com',
),
}
# my proposal
new_hosts_2 = {
v: k
for k, v_list in new_hosts.items()
for v in v_list}
print(new_hosts_2)
# {
# '1.oldhost-E.com': 'newhost-E.com',
# '2.oldhost-E.com': 'newhost-E.com',
# '1.oldhost-A.com': 'newhost-A.com',
# '2.oldhost-A.com': 'newhost-A.com',
# '3.oldhost-A.com': 'newhost-A.com',
# }
This does repeat the new host names (the values in new_hosts_2), but it will allow you to quickly look up given an old host name:
some_old_host = 'x.oldhost.com'
the corresponding_new_host = new_hosts_2[some_old_host]
Now you just need to:
read the lines of the file
find the old hostname in that line
lookup the corresponding new host in new_hosts_2
replace that value in the line
write the line to a new file
Maybe like this:
with open(file_name_1, 'r') as fr:
with open(file_name_2, 'w') as fw:
for line in fr:
line = line.strip()
if len(line) > 0:
# logic to find the start and end position of the old host
start_i = ?
end_i = ?
# get and replace, but only if its found in 'new_hosts_2'
old_host = line[start_i:end_i]
if old_host in new_hosts_2:
line = line[:start_i] + new_hosts_2[old_host] + line[end_i:]
fw.write(line + '\n')

Thank you for your tips. I came up with this now and it is working fine.
import fileinput
textfile = 'somefile.txt'
curhost = 'newhost.com'
hostlist = {curhost: ('1.oldhost.com',
'2.oldhost.com',
'3.oldhost.com')
}
new_hosts_2 = {
v: k
for k, v_list in hostlist.items()
for v in v_list}
for line in fileinput.input(textfile, inplace=True):
line = line.rstrip()
if not line:
continue
for f_key, f_value in new_hosts_2.items():
if f_key in line:
line = line.replace(f_key, f_value)
print line

If statement search for string returns false but printed output displays string

I'm using this function to search a site to see if a particular item i'm interested in s on sale. It first grabs the html from the page, then searches for an item i'm interested. When it finds the item it adds a number of the following lines (dictated by the rangenumber) to the variable 'endresult'. It then searches for the keyword ("sale") in endresult, at which point I'd like it to notify me if the keyword is present or not.
When I print endresult the output contains the keyword, but the if statement at the very end of the function always returns "keyword is missing" despite this and I can't work out why.
def bargainscraper(self, website, item, keyword,rangenum):
request = urllib.request.Request(website)
response = urllib.request.urlopen(request)
data = response.read()
html = str(data)
data1 = html2text.html2text(html)
fw = open('result1.txt', 'w')
fw.write(str(data1))
fw.close()
with open('result1.txt', 'r') as f:
for line in f:
if item in line:
for x in range(rangenum):
endresult = str(f.readline())
print (endresult)
if keyword in endresult:
print("keyword is present")
else:
print("keyword is missing")

Possibly need to concatenate the endresult instead of overwriting it with something like: endresult += str(f.readline()) notice the "+" before the "=".

I found writing the endresult to a file within the for loop then searching that file for the keyword outside of the for loop was the answer I was looking for:
def bargainscraper(self, website, item, keyword,rangenum):
request = urllib.request.Request(website)
response = urllib.request.urlopen(request)
data = response.read()
html = str(data)
data1 = html2text.html2text(html)
fw = open('result1.txt', 'w')
fw.write(str(data1))
fw.close()
with open('result1.txt', 'r') as f:
for line in f:
if item in line:
for x in range(rangenum):
endresult = str(f.readline())
# the 'a' switch is used to append
with open('result2.txt', 'a') as n:
n.write(endresult)
# This is outside of the for loop as otherwise it will iterate for each line of the rangenum
if keyword in open('result2.txt').read():
print ("keyword is present")
else:
print ("keyword is missing")

Remove duplicates after altering items

I have a script to clean urls to get base domains from example.com/example1 and example.com/example2 down to example.com My issue is when it goes to through the file of urls it will have duplicate base domains. I want to remove the duplicates while printing the urls to a file. below is the code I currently have.
enter from Tkinter import *
import tkFileDialog
import re
def main():
fileOpen = Tk()
fileOpen.withdraw() #hiding tkinter window
file_path = tkFileDialog.askopenfilename(
title="Open file", filetypes=[("txt file",".txt")])
if file_path != "":
print "you chose file with path:", file_path
else:
print "you didn't open anything!"
fin = open(file_path)
fout = open("URL Cleaned.txt", "wt")
for line in fin.readlines():
editor = (line.replace('[.]', '.')
.replace('[dot]', '.')
.replace('hxxp://www.', '')
.replace('hxxps://www.', '')
.replace('hxxps://', '')
.replace('hxxp://', '')
.replace('www.', '')
.replace('http://www.', '')
.replace('https://www.', '')
.replace('https://', '')
.replace('http://', ''))
editor = re.sub(r'/.*', '', editor)
if __name__ == '__main__':
main()
Any help is appreciated. I have scoured the posts and tried all of the suggestions for my issue and have not found one that works.

You can use regular expresion to find the base domains.
If you have one url per line in your file:
import re
def main():
file = open("url.txt",'r')
domains = set()
# will works for any web like https://www.domain.com/something/somethingmore... , also without www, without https or just for www.domain.org
matcher= re.compile("(h..ps?://)?(?P<domain>(www\.)?[^/]*)/?.*")
for line in file:
# make here any replace you need with obfuscated urls like: line = line.replace('[.]','.')
if line[-1] == '\n': # remove "\n" from end of line if present
line = line[0:-1]
match = matcher.search(line)
if match != None: # If a url has been found
domains.add(match.group('domain'))
print domains
file.close()
main()
For example, with this file, it will print:
set(['platinum-shakers.net', 'wmi.ns01.us', 'adservice.no-ip.org', 'samczeruno.pl', 'java.ns1.name', 'microsoft.dhcp.biz', 'ids.us01.us', 'devsite.quostar.com', 'orlandmart.com'])

perhaps you could use a regular expression:
import re
p = re.compile(r".*\.com/(.*)") # to get for instance 'example1' or 'example2' etc.
with open(file_path) as fin, open("URL Cleaned.txt", "wt") as fout:
lines = fin.readlines():
bases = set(re.search(p, line).groups()[0] for line in lines if len(line) > 1)
for b in bases:
fout.write(b)
Using with open(..) auto closes the files after the executing the block of code
Output:
Using a text file with:
www.example.com/example1
www.example.com/example2
# blank lines are accounted for
www.example.com/example3
www.example.com/example4
www.example.com/example4 # as are duplicates
as the lines, I got the output,
example1
example2
example3
example4