Develop Reference

Why are Lists causing problems

So I am working on a certain code to modify a text file. When I use this function individually, it works perfectly
TextRotation.rotTextC("cv.txt")
But when I use it in batch as a list like this
def files_LTXT(pathF):
return glob.glob(pathF + "*" + ".txt")
for i in range (len(listFileTXT)):
TextRotation.rotTextC(listFileTXT[i])
IT gives the following error:
File "C:\Users\Administrator\PycharmProjects\openCV\TextRotation.py", line
9, in rotLineC
0
valueObj = int(lineStr[c1])
0.472917 0.713281 0.845833 0.376563
IndexError: string index out of range
Function rotLineC is as follows:
def rotLineC(lineStr, c1):
if len(lineStr) > 2:
valueObj = int(lineStr[c1])
print(valueObj)
valueXC = float(lineStr[(c1+2):(c1+10)])
valueYC = float(lineStr[(c1+11):(c1+19)])
valueW = float(lineStr[(c1+20):(c1+28)])
valueH = float(lineStr[(c1+29):(c1+37)])
# print(valueXC)
# print(valueYC)
# print(valueW)
# print(valueH)
nValueXC = round(1 - valueYC, 6)
nValueYC = round(valueXC, 6)
nValueW = round(valueH, 6)
nValueH = round(valueW, 6)
rotString = str(int(valueObj)) + " " + str(nValueXC) + " " + \
str(nValueYC) + " " + str(nValueW) + " " + str(nValueH)
print(str(nValueXC) + " " + str(nValueYC) + " " + str(nValueW) + " " + str(nValueH))
print(rotString)
return rotString
This function works fine!
for i in range (len(listFileJPG)):
ImageRotation.rotImage(listFileJPG[i])

Mind to include the / to the end of the path! (I am assuming a UNIX environment here)
If the path is 'dev/my_pat', for example, your function will fail. The path must end with a /. You can it to your function:
...
if pathF[-1] != '/':
return glob.glob(pathF + "/*.txt")
...
Also, do not iterate using indices, use the pythonic way!
for file in listFileTXT(my_path):
TextRotation.rotTextC(file)

how do I find a way to use ssh and rsync

I have troubles using to implement ssh and rsync including a private key in python, including Popen (subprocess).
Basically the rsync syntax to use should be:
$ rsync -az -e --log-file=$logdir/logfile.out \
'ssh -e /home/user/.ssh/id_rsa' user#server:/target-directory
What I have is this:
import subprocess
First, I build my logdir path - with variables :
logdir = [basedir + '/' + 'log' + '/' + today + '/' + 'pfdcopy_' \
+ typ + '_' + ts + '.log.txt']
Then I build the target directory:
target= ['jboss' + '#' + targetsvr + ':' + /data']
Finally, I try to run this code
p1 = subprocess.Popen(['rsync', '-az', '--log-file=%s' % \
logdir/logfile.out , '-e', 'ssh', '-i', \
'/home/user/.ssh/id_rsa', target])
It's quite complex, I know, mainly because of the variables, and the quotation marks.
Running this, I get always different syntax errors with p1.
Any help is highly appreciated. Thanks!
edited (08-10-2018):
here is my complete runnable code snippet -
from datetime import datetime
import subprocess
import os
import fnmatch
now = datetime.now()
today = now.strftime("%Y-%m-%d")
ts = now.strftime("%Y-%m-%d-%H-%M-%S")
sign_output_dir = '/Users/fanta4/Documents/python-files/outgoing'
mandator = 'BTV'
formsize = 'A4'
basedir = '/Users/fanta4/Documents'
pdf_to_send = []
targetsvr = 'nas1'
doktyp = (
'WPGEBUEHR', 'WPDURCHFU', 'WPABR', 'WPABRKF', 'WPABRVK', 'WPABRTILG', 'WPABRERTR', 'WPAMIS', 'WPSTREP',
'WPABLAUF', 'WPAVISO', 'WPAUSZUG', 'WPERTRAEG', 'WPSIKTEST', 'WPTRANS', 'WPANSCHAFF', 'KKKONTOMIT', 'KRKURSUEW',
'WPVERLUSTA', 'WPVERLUSTG')
os.chdir(sign_output_dir)
for file in os.listdir(sign_output_dir):
if fnmatch.fnmatch(file, '*.pdf'):
pdf_to_send.append(file)
os.chdir(sign_output_dir)
print('debug: doktyp ist: {}'.format(formsize))
for typ in doktyp:
if typ in str(pdf_to_send):
ts = now.strftime("%Y-%m-%d-%Hh-%Mm-%Ss")
print('typ: {:12s} exists -> will be transfered to nas1'.format(typ))
logdir = [basedir + '/' + 'log' + '/' + mandator + '/' + today + '/' + 'pfdcopy_' + typ + '_' + ts + '.log.txt']
target = ['jboss' + '#' + targetsvr + '/data' + '/' + mandator + typ]
p1 = subprocess.Popen(
['rsync', '-az', '--log-file=%s' % logdir, '-e', 'ssh', '-i', '/Users/fanta4/.ssh/id_rsa', typ, '-l', target])
p1.returncode
if p1 > 0:
print('debug: Error with rsync of typ: {} to target: {}'.format(typ, targetsvr))
else:
print('debug: rsync mandator: {:3s} with typ: {:12s} succeeded'.format(mandator, typ))
else:
print('debug: typ: {:12s} does not exist'.format(typ))
logfile = ['/data' + '/' + 'log' + '/' + mandator + '/' + ts]
print('debug: pls see logfile in: {}'.format(logfile))
If I run this code, I get:
/Users/fanta4/anaconda3/bin/python "/Users/fanta4/Library/Mobile Documents/com~apple~CloudDocs/entw/python/prog/rsync-test.py"
Traceback (most recent call last):
/Users/fanta4/Documents/python-files/outgoing
debug: doktyp ist: A4
File "/Users/fanta4/Library/Mobile
Documents/com~apple~CloudDocs/entw/python/prog/rsync-test.py", line 37, in <module>
typ: WPGEBUEHR exists -> will be transfered to nas1
['rsync', '-az', '--log-file=%s' % logdir, '-e', 'ssh', '-i', '/Users/fanta4/.ssh/id_rsa', typ, '-l', target])
File "/Users/fanta4/anaconda3/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/Users/fanta4/anaconda3/lib/python3.6/subprocess.py", line 1275, in _execute_child
restore_signals, start_new_session, preexec_fn)
TypeError: expected str, bytes or os.PathLike object, not list
Process finished with exit code 1

Your issue is exemplified by the following lines (when used to generate items which are later used as elements inside an argument vector):
logdir = [basedir + '/' + 'log' + '/' + mandator + '/' + today + '/' + 'pfdcopy_' + typ + '_' + ts + '.log.txt']
target = ['jboss' + '#' + targetsvr + '/data' + '/' + mandator + typ]
You're defining logdir and target as lists (with only one string inside them), whereas they need to be strings.
Just take out the square brackets that create a list, and you'll have strings instead:
logdir = basedir + '/log/' + mandator + '/' + today + '/pfdcopy_' + typ + '_' + ts + '.log.txt'
target = 'jboss#' + targetsvr + '/data/' + mandator + typ

You haven't mentioned what syntax errors you are getting. It would really for the benefit of everyone for you to include that information. I am guessing it's missing the missing quotes around one the string parameters.
p1 = subprocess.Popen([
'rsync', '-az', '--log-file=%s' % 'logdir/logfile.out',
'-e', 'ssh', '-i',
'/home/user/.ssh/id_rsa', target
])

syntax error for writing a variable in a script

dump.pbd='pdb' + pdbFile + '_' + 'res' + residMin + '_' residMax + '.pdb'
the program keep giving me syntax error when I run it.
import re
import sys
import os
import time
from sys import argv
import xmltodict
if len(sys.argv) < 3:
message = '\n Get protein file in the form of pdf file from pdb website. \n\n Usage: '+sys.argv[0] + ' [4-letter PDBid] [resid range] \n' + ' Example: ' + sys.argv[0] + ' 2rh1 53-71\n' + ' Output File: pdb2rh1_res53-71.pdb'
print (message)
exit()
pdbID=sys.argv[1]
residRange=sys.argv[2]
residData=residRange.split('-')
residMin=int(residData[0])
residMax=int(residData[1])
twoletter=pdbID[1:3]
xmlfile=pdbID + '.xml'
pdbgzfile=pdbID + '.pdb.gz'
pdbFile=pdbID+'.pdb'
dump.pbd='pdb' + pdbFile + '_' + 'res' + residMin + '_' residMax + '.pdb'
wgetcom='wget https://files.rcsb.org/view/'+pdbFile+' -O '+pdbFile
print(wgetcom)
os.system(wgetcom)
f = open (pdbFile,'r')
k = 0
rc = 0
data = f.readlines()
g = open (dump.pdb, 'w')
for linedata in data:
line=linedata.strip()
words = line.split()
if(words[0] == 'ATOM'):
k=k+1
words[5]=int(line[22:26].strip())
if(words[5] in range(residMin,residMax+1)):
g.write(linedata)
for i in words:
if(i=='CA'):
rc = rc+1
print(rc)
the code is not working because it is giving me a syntax error for line number 22 that states dump.pbd='pdb' + pdbFile + '' + 'res' + residMin + '' residMax + '.pdb'. so can you please help me with that?
Thanks so much on advance!

You've forgotten to add a + sign.
This line should work: dump.pbd='pdb' + pdbFile + '' + 'res' + residMin + '' + residMax + '.pdb'
There must be a + sign between '_' and residMax as this is the Python concatenating strings syntax.

Python Permission Error [WinError32]...being used by another process

I have some images in a folder and if their dimensions are not what I want, I copy them to another folder and then resize them using the thumbnail command. After I've done that I want to rename them and as part of their new name I want to include the pixel sizes.
Anyway I've tried my code and I run into the following error:
File "C:/Anaconda/PhD/Scikit-Learn/Image Resizing3.py", line 55, in resize_img
os.rename(image_directory + '/Resized'+ "/" + image_filenames[i],image_directory + '/Resized'+ "/" + image_filenames[i][0:(len(image_filenames[i])-4)]+ str(img1.size[0]) + ' x ' + str(img1.size[1]) + '.jpg' )
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process:
I thought if I used the with keyword that it would close any processes I was using.
If anyone can see any problems in my code wrt to the error then I'd really appreciate it if you have a solution for it.
import scipy.io as sio
from matplotlib import pyplot as plt
from matplotlib import image as mpimg
from PIL import Image
from collections import OrderedDict
import shutil as sh
import os
image_directory = 'C:/StreetView/Labelled Images with Classes/Folder1 /Labelled/Classes/wood'
def resize_img(image_directory, pixel_width, pixel_height):
image_filenames = []
DIR = os.listdir(image_directory)
for n in DIR:
if n.endswith('.jpg'):
image_filenames.append(n)
image_sizes = []
images = []
for i in range(len(image_filenames)):
with Image.open(image_directory + '/' + image_filenames[i]) as images:
image_sizes.append(tuple(images.size))
ordered_imsize = tuple(OrderedDict.fromkeys(image_sizes))
pixarea = []
for i in range(len(ordered_imsize)):
arg1 = ordered_imsize[i][0]
arg2 = ordered_imsize[i][1]
pixarea.append(arg1 * arg2)
print('The maximum pixel area is', max(pixarea))
print('The image dimensions giving the largest area is:', ordered_imsize[pixarea.index(max(pixarea))] )
print('The minimum pixel area is', min(pixarea))
print('The image dimensions giving the smallest area is:', ordered_imsize[pixarea.index(min(pixarea))] )
if not os.path.exists(image_directory + '/Resized'):
os.mkdir(image_directory + '/Resized') # Then make the folder
for i in range(len(image_directory)):
if (image_sizes[i][0] >= 100) or (image_sizes[i][1] >= 400):
print('Inside the greater than loop')
sh.copy(image_directory + "/" + image_filenames[i],image_directory + '/Resized')
image_sizes1 = []
with Image.open(image_directory + '/Resized'+ "/" + image_filenames[i]) as img1: # need to use the with keyword because this makes sure the file is closed after its been used so it doesn't cause any conflicts like it did when it wasn't used.
print('Checking size: ', img1.size)
img2 = img1.thumbnail((pixel_width,pixel_height)) # img1 is not the thumbnail
image_sizes1.append(tuple(img1.size))
os.rename(image_directory + '/Resized'+ "/" + image_filenames[i],image_directory + '/Resized'+ "/" + image_filenames[i][0:(len(image_filenames[i])-4)]+ str(img1.size[0]) + ' x ' + str(img1.size[1]) + '.jpg' )
print('Image number', i, 'has been resized')
else:
print('inside the else loop')
sh.copy(image_directory + "/" + image_filenames[i],image_directory + '/Resized')
image_sizes2 = []
with Image.open(image_directory + '/Resized'+ "/" + image_filenames[i]) as img3:
image_sizes2.append(tuple(img3.size))
img3.save(image_directory + '/Resized' + "/" + image_filenames[i][0:len(image_filenames[i])-4] + '_' + str(img3.size[0]) + 'x' + str(img3.size[1]) + ".jpg")
#os.remove(image_directory + '/Resized'+ "/" + image_filenames[i])
print('Image number', i, 'has been copied over and not resized')
resize_img(image_directory,100,400)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process

Solved it, I used shutil.move instead of os.rename.

Script Loop through files in directory

I have the following code which creates the txt file I require from a shp.file with the data I need. I have a folder called profiles containing a few number of shape files named (profil1.shp, profil2.shp, profil3.shp etc.). I was wondering how to create a loop so that the script creates for each file a txt file with the same name (eg. for profil1.shp create profil1.txt, profil2.shp create profil2.txt and so on).
import ogr, os, sys, osr
os.chdir('..\profiles')
file = open('profil1.txt', 'w')
driver = ogr.GetDriverByName('ESRI Shapefile')
datasource = driver.Open('profil1.shp', 0)
if datasource is None:
print 'Could not open file'
sys.exit(1)
layer = datasource.GetLayer()
feature = layer.GetNextFeature()
while feature:
id = feature.GetFieldAsString('ID')
Distanta = feature.GetFieldAsString('DIST')
Z = feature.GetFieldAsString('Z')
geom = feature.GetGeometryRef()
x = str(geom.GetX())
y = str(geom.GetY())
file.write(id + " " + Distanta + " " + "[X]:" + " " + x + ' ' + '[Y]:' + " " + y + " " + " " + "[Z]" + Z + " " + "\n")
feature.Destroy()
feature = layer.GetNextFeature()
datasource.Destroy()
file.close()
edit: the code is returning a Could not open file.Photo of the folder containing the files and their respective names. Safe to assume I am doing something wrong.
import ogr, os, sys, osr,os.path
os.chdir = ('C:\Users\Andrei\Desktop\profil3')
l = os.listdir('C:\Users\Andrei\Desktop\profil3')
for i in l:
if i.endswith('.shp'):
s1 = s.split('.')[0] + '.txt'
file = open(s1, 'w')
driver = ogr.GetDriverByName('ESRI Shapefile')
datasource = driver.Open(i, 0)
if datasource is None:
print 'Could not open file'
sys.exit(1)
layer = datasource.GetLayer()
feature = layer.GetNextFeature()
while feature:
id = feature.GetFieldAsString('ID')
Distanta = feature.GetFieldAsString('DIST')
Z = feature.GetFieldAsString('Z')
geom = feature.GetGeometryRef()
x = str(geom.GetX())
y = str(geom.GetY())
file.write(id + " " + Distanta + " " + "[X]:" + " " + x + ' ' + '[Y]:' + " " + y + " " + " " + "[Z]" + Z + " " + "\n")
feature.Destroy()
feature = layer.GetNextFeature()
datasource.Destroy()
file.close()

You can use os.listdir() to list the files and folders in the current directory.
This returns a list of all files in the current directory (or the directory given to it as parameter , if no parameter is specified it checks the current directory) .
Then you can check for files with the name ending with .shp using string.endswith() function and then use that to create your new files.
Example of a small portion -
import os , os.path
l = os.listdir()
for i in l:
if i.endswith('.shp'):
s1 = s.split('.')[0] + '.txt'
At the end s1 would contain the file with extension as .txt .
Then you can do your logic on this file, and keep on doing like this.
Full code would look something like -
import ogr, os, sys, osr,os.path
os.chdir('..\profiles')
l = os.listdir()
for i in l:
if i.endswith('.shp'):
s1 = s.split('.')[0] + '.txt'
file = open(s1, 'w')
driver = ogr.GetDriverByName('ESRI Shapefile')
datasource = driver.Open(i, 0)
if datasource is None:
print 'Could not open file'
sys.exit(1)
layer = datasource.GetLayer()
feature = layer.GetNextFeature()
while feature:
id = feature.GetFieldAsString('ID')
Distanta = feature.GetFieldAsString('DIST')
Z = feature.GetFieldAsString('Z')
geom = feature.GetGeometryRef()
x = str(geom.GetX())
y = str(geom.GetY())
file.write(id + " " + Distanta + " " + "[X]:" + " " + x + ' ' + '[Y]:' + " " + y + " " + " " + "[Z]" + Z + " " + "\n")
feature.Destroy()
feature = layer.GetNextFeature()
datasource.Destroy()
file.close()
A better way of openning files, etc is using with statement. Look up its tutorial here.