I'm trying to read in a list of account numbers, then have my program do a search in the appropriate directory for each account number. I want to capture the information from this search, to then split out the file name, date, and time as the output from my program. Currently I'm receiving this error: TypeError: bufsize must be an integer
Here is my code:
def app_files(level):
piv_list_file = raw_input(r"Please enter the full path of the file containing the Pivot ID's you would like to check: ")
piv_id_list = []
proc_out_list = []
input_dir = ''
try:
with open(piv_list_file, 'rbU') as infile:
for line in infile:
line = line.rstrip('\r\n')
piv_id_list.append(line)
except IOError as e:
print 'Unable to open the account number file: %s' % e.strerror
if level == 'p':
input_dir = '[redacted]'
else:
input_dir = '[redacted]'
subprocess.call('cd', input_dir)
for i, e in enumerate(piv_id_list):
proc_out = subprocess.check_output('ls', '-lh', '*CSV*APP*{0}.zip'.format(e))
proc_out_list.append(proc_out)
print(proc_out)
Your subprocess.check_output() function call is wrong. You should provide the complete command as a list (as the first argument). Example -
subprocess.check_output(['ls', '-lh', '*CSV*APP*{0}.zip'.format(e)])
Similar issue with subprocess.call in your code .
Related
I am writing a Python script for use by multiple non-Python users.
I have a text file containing the parameters my script needs to run.
One of the inputs is a path. I cannot get my script to run and was thinking it was because I had referenced my path incorrectly.
I have tried:
C:\temp\test
"C:\temp\test"
r"C:\temp\test"
C:/temp/test
"C:/temp/test"
C:\\temp\\test
"C:\\temp\\test"
I have added each one of these into a text file, which is called and read in my Python script.
I have other parameters and they are called correctly, my script seems to run when I hard code the path in. I say seems because I think there are a few bugs I need to check, but it runs with no errors.
When I use the text file I get this error - which varies depending on if I used one of the above examples:
WindowsError: [Error 123] The filename, directory name, or volume
label syntax is incorrect: 'c:\temp\match1\jpg\n/.'
My code is as follows:
print ("Linking new attachments to feature")
fp = open(r"C:\temp\Match1\Match_Table.txt","r") #reads my text file with inputs
lines=fp.readlines()
InFeat = lines[1]
print (InFeat)
AttFolder = lines[3] #reads the folder from the text file
print (AttFolder)
OutTable = lines[5]
if arcpy.Exists(OutTable):
print("Table Exists")
arcpy.Delete_management(OutTable)
OutTable = lines[5]
print (OutTable)
LinkF = lines[7]
print (LinkF)
fp.close()
#adding from https://community.esri.com/thread/90280
if arcpy.Exists("in_memory\\matchtable"):
arcpy.Delete_management("in_memory\\matchtable")
print ("CK Done")
input = InFeat
inputField = "OBJECTID"
matchTable = arcpy.CreateTable_management("in_memory", "matchtable")
matchField = "MatchID"
pathField = "Filename"
print ("Table Created")
arcpy.AddField_management(matchTable, matchField, "TEXT")
arcpy.AddField_management(matchTable, pathField, "TEXT")
picFolder = r"C:\temp\match1\JPG" #hard coded in
print (picFolder)
print ("Fields added")
fields = ["MatchID", "Filename"]
cursor = arcpy.da.InsertCursor(matchTable, fields)
##go thru the picFolder of .png images to attach
for file in os.listdir(picFolder):
if str(file).find(".jpg") > -1:
pos = int(str(file).find("."))
newfile = str(file)[0:pos]
cursor.insertRow((newfile, file))
del cursor
arcpy.AddAttachments_management(input, inputField, matchTable, matchField, pathField, picFolder)
From your error "'c:\temp\match1\jpg\n/.'", i can see "\n" character, \n is symbole of new line ( when you press enter button ) you should remove that character from end of your path! did you try to do that? you can use .lstrip("\n") , replcae() or regx methods for remove that character.
Try to open and read line by line of your input file like this:
read_lines = [line.rstrip('\n') for line in open(r"C:\temp\Match1\Match_Table.txt")]
print(read_lines)
print(read_lines[1])
I've never used Python and have copied some script (with permission) from someone online, so I'm not sure why the code is dropping. I'm hoping someone can understand it and put it right for me!
from os import walk
from os.path import join
#First some options here.
!RootDir = "C:\\Users\\***\\Documents\\GoGames"
!OutputFile = "C:\\Users\\***\\Documents\\GoGames\\protable.csv"
Properties = !!['pb', 'pw', 'br', 'wr', 'dt', 'ev', 're']
print """
SGF Database Maker
==================
Use this program to create a CSV file with sgf info.
"""
def getInfo(filename):
"""Read out file info here and return a dictionary with all the
properties needed."""
result = !![]
file = open(filename, 'r')
data = file.read(1024) read at most 1kb since we assume all relevant info is in the beginning
file.close()
for prop in Properties:
try:
i = data.lower().index(prop)
except !ValueError:
result.append((prop, ''))
continue
try:
value = data![data.index('![', i)+1 : data.index(']', i)]
except !ValueError:
value = ''
result.append((prop, value))
return dict(result)
!ProgressCounter = 0
file = open(!OutputFile, "w")
file.write('^Filename^;^PB^;^BR^;^PW^;^WR^;^RE^;^EV^;^DT^\n')
for root, dirs, files in walk(!RootDir):
for name in files:
if name![-3:].lower() != "sgf":
continue
info = getInfo(join(root, name))
file.write('^'+join(root, name)+'^;^'+info!['pb']+'^;^'+info!['br']+'^;^'+info!['pw']+'^;^'+info!['wr']+'^;^'+info!['re']+'^;^'+info!['ev']+'^;^'+info!['dt']+'^\n')
!ProgressCounter += 1
if (!ProgressCounter) % 100 == 0:
print str(!ProgressCounter) + " games processed."
file.close()
print "A total of " + str(!ProgressCounter) + " have been processed."
Using Netbeans IDE I get the following error:
!RootDir = "C:\\Users\\***\\Documents\\GoGames"
^
SyntaxError: mismatched input '' expecting EOF
I have previously been able to step through the code as far as file.close(), where I go an error "does not match outer indentation level".
Anyone able to put the syntax of this code right for me?
Remove the exclamation marks in front of variable names, list declarations (!![]) and in except clauses (except !ValueError), this is not valid Python syntax.
I am stuck why the words.txt is not showing the full grid, below is the tasks i must carry out:
write code to prompt the user for a filename, and attempt to open the file whose name is supplied. If the file cannot be opened the user should be asked to supply another filename; this should continue until a file has been successfully opened.
The file will contain on each line a row from the words grid. Write code to read, in turn, each line of the file, remove the newline character and append the resulting string to a list of strings.After the input is complete the grid should be displayed on the screen.
Below is the code i have carried out so far, any help would be appreciated:
file = input("Enter a filename: ")
try:
a = open(file)
with open(file) as a:
x = [line.strip() for line in a]
print (a)
except IOError as e:
print ("File Does Not Exist")
Note: Always avoid using variable names like file, list as they are built in python types
while True:
filename = raw_input(' filename: ')
try:
lines = [line.strip() for line in open(filename)]
print lines
break
except IOError as e:
print 'No file found'
continue
The below implementation should work:
# loop
while(True):
# don't use name 'file', it's a data type
the_file = raw_input("Enter a filename: ")
try:
with open(the_file) as a:
x = [line.strip() for line in a]
# I think you meant to print x, not a
print(x)
break
except IOError as e:
print("File Does Not Exist")
You need a while loop?
while True:
file = input("Enter a filename: ")
try:
a = open(file)
with open(file) as a:
x = [line.strip() for line in a]
print (a)
break
except IOError:
pass
This will keep asking untill a valid file is provided.
For a class assignment, I'm supposed to grab the contents of a file, compute the MD5 hash and store it in a separate file. Then I'm supposed to be able to check the integrity by comparing the MD5 hash. I'm relatively new to Python and JSON, so I thought I'd try to tackle those things with this assignment as opposed to going with something I already know.
Anyway, my program reads from a file, creates a hash, and stores that hash into a JSON file just fine. The problem comes in with my integrity checking. When I return the results of the computed hash of the file, it's different from what is recorded in the JSON file even though no changes have been made to the file. Below is an example of what is happening and I pasted my code as well. Thanks in advance for the help.
For example: These are the contents of my JSON file
Content: b'I made a file to test the md5\n'
digest: 1e8f4e6598be2ea2516102de54e7e48e
This is what is returned when I try to check the integrity of the exact same file (no changes made to it):
Content: b'I made a file to test the md5\n'
digest: ef8b7bf2986f59f8a51aae6b496e8954
import hashlib
import json
import os
import fnmatch
from codecs import open
#opens the file, reads/encodes it, and returns the contents (c)
def read_the_file(f_location):
with open(f_location, 'r', encoding="utf-8") as f:
c = f.read()
f.close()
return c
def scan_hash_json(directory_content):
for f in directory_content:
location = argument + "/" + f
content = read_the_file(location)
comp_hash = create_hash(content)
json_obj = {"Directory": argument, "Contents": {"filename": str(f),
"original string": str(content), "md5": str(comp_hash)}}
location = location.replace(argument, "")
location = location.replace(".txt", "")
write_to_json(location, json_obj)
#scans the file, creates the hash, and writes it to a json file
def read_the_json(f):
f_location = "recorded" + "/" + f
read_json = open(f_location, "r")
json_obj = json.load(read_json)
read_json.close()
return json_obj
#check integrity of the file
def check_integrity(d_content):
#d_content = directory content
for f in d_content:
json_obj = read_the_json(f)
text = f.replace(".json", ".txt")
result = find(text, os.getcwd())
content = read_the_file(result)
comp_hash = create_hash(content)
print("content: " + str(content))
print(result)
print(json_obj)
print()
print("Json Obj: " + json_obj['Contents']['md5'])
print("Hash: " + comp_hash)
#find the file being searched for
def find(pattern, path):
result = ""
for root, dirs, files in os.walk(path):
for name in files:
if fnmatch.fnmatch(name, pattern):
result = os.path.join(root, name)
return result
#create a hash for the file contents being passed in
def create_hash(content):
h = hashlib.md5()
key_before = "reallyBad".encode('utf-8')
key_after = "hashKeyAlgorithm".encode('utf-8')
content = content.encode('utf-8')
h.update(key_before)
h.update(content)
h.update(key_after)
return h.hexdigest()
#write the MD5 hash to the json file
def write_to_json(arg, json_obj):
arg = arg.replace(".txt", ".json")
storage_location = "recorded/" + str(arg)
write_file = open(storage_location, "w")
json.dump(json_obj, write_file, indent=4, sort_keys=True)
write_file.close()
#variable to hold status of user (whether they are done or not)
working = 1
#while the user is not done, continue running the program
while working == 1:
print("Please input a command. For help type 'help'. To exit type 'exit'")
#grab input from user, divide it into words, and grab the command/option/argument
request = input()
request = request.split()
if len(request) == 1:
command = request[0]
elif len(request) == 2:
command = request[0]
option = request[1]
elif len(request) == 3:
command = request[0]
option = request[1]
argument = request[2]
else:
print("I'm sorry that is not a valid request.\n")
continue
#if user inputs command 'icheck'...
if command == 'icheck':
if option == '-l':
if argument == "":
print("For option -l, please input a directory name.")
continue
try:
dirContents = os.listdir(argument)
scan_hash_json(dirContents)
except OSError:
print("Directory not found. Make sure the directory name is correct or try a different directory.")
elif option == '-f':
if argument == "":
print("For option -f, please input a file name.")
continue
try:
contents = read_the_file(argument)
computedHash = create_hash(contents)
jsonObj = {"Directory": "Default", "Contents": {
"filename": str(argument), "original string": str(contents), "md5": str(computedHash)}}
write_to_json(argument, jsonObj)
except OSError:
print("File not found. Make sure the file name is correct or try a different file.")
elif option == '-t':
try:
dirContents = os.listdir("recorded")
check_integrity(dirContents)
except OSError:
print("File not found. Make sure the file name is correct or try a different file.")
elif option == '-u':
print("gonna update stuff")
elif option == '-r':
print("gonna remove stuff")
#if user inputs command 'help'...
elif command == 'help':
#display help screen
print("Integrity Checker has a few options you can use. Each option "
"must begin with the command 'icheck'. The options are as follows:")
print("\t-l <directory>: Reads the list of files in the directory and computes the md5 for each one")
print("\t-f <file>: Reads a specific file and computes its md5")
print("\t-t: Tests integrity of the files with recorded md5s")
print("\t-u <file>: Update a file that you have modified after its integrity has been checked")
print("\t-r <file>: Removes a file from the recorded md5s\n")
#if user inputs command 'exit'
elif command == 'exit':
#set working to zero and exit program loop
working = 0
#if anything other than 'icheck', 'help', and 'exit' are input...
else:
#display error message and start over
print("I'm sorry that is not a valid command.\n")
Where are you defining h, the md5 object being used in this method?
#create a hash for the file contents being passed in
def create_hash(content):
key_before = "reallyBad".encode('utf-8')
key_after = "hashKeyAlgorithm".encode('utf-8')
print("Content: " + str(content))
h.update(key_before)
h.update(content)
h.update(key_after)
print("digest: " + str(h.hexdigest()))
return h.hexdigest()
My suspicion is that you're calling create_hash twice, but using the same md5 object in both calls. That means the second time you call it, you're really hashing "reallyBad*file contents*hashkeyAlgorithmreallyBad*file contents*hashKeyAlgorithm". You should create a new md5 object inside of create_hash to avoid this.
Edit: Here is how your program runs for me after making this change:
Please input a command. For help type 'help'. To exit type 'exit'
icheck -f ok.txt Content: this is a test
digest: 1f0d0fd698dfce7ce140df0b41ec3729 Please input a command. For
help type 'help'. To exit type 'exit' icheck -t Content: this is a
test
digest: 1f0d0fd698dfce7ce140df0b41ec3729 Please input a command. For
help type 'help'. To exit type 'exit'
Edit #2:
Your scan_hash_json function also has a bug at the end of it. You're removing the .txt suffix from the file, and calling write_to_json:
def scan_hash_json(directory_content):
...
location = location.replace(".txt", "")
write_to_json(location, json_obj)
However, write_to_json is expecting the file to end in .txt:
def write_to_json(arg, json_obj):
arg = arg.replace(".txt", ".json")
If you fix that, I think it should do everything as expected...
I see 2 possible problems you are facing:
for hash computation is computing from a binary representation of a string
unless you work only with ASCII encoding, the same international character e.g. č has different representations in the UTF-8 or Unicode encoding.
To consider:
If you need UTF-8 or Unicode, normalize first your content before you save it or calculate a hash
For testing purposes compare content binary representation.
use UTF-8 only for IO operations, codecs.open does all conversion
for you
from codecs import open
with open('yourfile', 'r', encoding="utf-8") as f:
decoded_content = f.read()
Thanks to stackoverflow, I am able to read and copy a file. However, I need to read a picture file one line at a time, and the buffer array can't exceed 3,000 integers. How would I separate the lines, read them, and then copy them? Is that the best way to execute this?
Here is my code, courtesy of #Chayim:
import os
import sys
import shutil
import readline
source = raw_input("Enter source file path: ")
dest = raw_input("Enter destination path: ")
file1 = open(source,'r')
if not os.path.isfile(source):
print "Source file %s does not exist." % source
sys.exit(3)
file_line = infile.readline()
try:
shutil.copy(source, dest)
infile = open(source,'r')
outfile = open(dest,'r')
file_contents = infile.read()
file_contents2 = outfile.read()
print(file_contents)
print(file_contents2)
infile.close()
outfile.close()
except IOError, e:
print "Could not copy file %s to destination %s" % (source, dest)
print e
sys.exit(3)
I added
file_line = infile.readline()
but I'm concerned that infile.readline() will return a string, instead of integers. Also, how do I limit the number of integers it processes?
I think you want to do something like this:
infile = open(source,'r')
file_contents_lines = infile.readlines()
for line in file_contents_lines:
print line
This will get you all the lines in the file and put them into a list containing each line as an element in the list.
Take a look at the documentation here.