I hope that I can ask this in a clear way, im very much a beginner to python and forums in general so I apologise if i've got anything wrong from the start!
My issue is that I am currently trying to use os.system() to enable a program to run on every file within a directory (this is a directory of ASCII tables which I am crossing with a series of other tables to find matches.
import os
for filename in os.listdir('.'):
os.system('stilts tmatch2 ifmt1=ascii ifmt2=ascii in1=intern in2= %s matcher=2d values1='col1 col2' values2='col1 col2' params=5 out= %s-table.fits'%(filename,filename))
So what im hoping this would do is for every 'filename' it would operate this program known as stilts. Im guessing this gets interrupted/doesn't work because of the presence of apostrophes ' in the line of code itself, which must disrupt the syntax? (please correct me if I am wrong)
I then replaced the ' in os.system() with "" instead. This, however, stops me using the %s notation to refer to filenames throughout the code (at least I am pretty sure anyway).
import os
for filename in os.listdir('.'):
os.system("stilts tmatch2 ifmt1=ascii ifmt2=ascii in1=intern in2= %s matcher=2d values1='col1 col2' values2='col1 col2' params=5 out= %s-table.fits"%(filename,filename))
This now runs but obviously doesn't work, as it inteferes with the %s input.
Any ideas how I can go about fixing this? are there any alternative ways to refer to all of the other files given by 'filename' without using %s?
Thanks in advance and again, sorry for my inexperience with both coding and using this forum!
I am not familiar with os.system() but maybe if you try do some changes about the string you are sending to that method before it could behave differently.
You must know that in python you can "sum" strings so you can save your commands in a variable and add the filenames as in:
os.system(commands+filename+othercommands+filename)
other problem that could be working is that when using:
for file in os.listdir()
you may be recievin file types instead of the strings of their names. Try using a method such as filename.name to check if this is a different type of thing.
Sorry I cant test my answers for you but the computer I am using is too slow for me to try downloading python.
Related
I'm trying to run an external program from a Python script.
After searching and reading multiple post here I came to what seemed to be the solution.
First, I used subprocess.call function.
If I build the command this way:
hmmer1=subprocess.call("D:\Python_Scripts\HMMer3\hmmsearch.exe --tblout hmmTestTab.out SDHA.hmm Test.fasta")
The external program D:\Python_Scripts\HMMer3\hmmsearch.exe is run taking hmmTestTab.out as file name for the output and SDHA.hmm and Test.fasta as input files.
Nevertheless, if I try to replace the file names with the variables outfile, hmmprofile and fastafile (I intend to receive those variables as arguments for the Python script and use them to build the external program call),
hmmer2=subprocess.call("D:\Python_Scripts\HMMer3\hmmsearch.exe --tblout outfile hmmprofile fastafile")
Python prints an error about being unable to open the input files.
I also used "Popen" function with analogous results:
This call works
hmmer3=Popen(['D:\Python_Scripts\HMMer3\hmmsearch.exe', '--tblout','hmmTestTab.out', 'SDHA.hmm','Test.fasta'])
and this one doesn't
hmmer4=Popen(['D:\Python_Scripts\HMMer3\hmmsearch.exe', '--tblout','outfile', 'hmmprofile','fastafile'])
As result of this, I presume I need to understand which is process to follow to interpolate the variables into the call, because it seems that the problem is there.
Would any of you help me with this issue?
Thanks in advance
You have:
hmmer4=Popen(['D:\Python_Scripts\HMMer3\hmmsearch.exe', '--tblout','outfile', 'hmmprofile','fastafile'])
But that's not passing the variable outfile. It's passing a string, 'outfile'.
You want:
hmmer4=Popen(['D:\Python_Scripts\HMMer3\hmmsearch.exe', '--tblout', outfile, hmmprofile, fastafile])
And the other answer is correct, though it addresses a different problem; you should double the backslashes, or use r'' raw strings.
Try to change this:
hmmer1=subprocess.call("D:\Python_Scripts\HMMer3\hmmsearch.exe"
to
hmmer1=subprocess.call('D:\\Python_Scripts\\HMMer3\\hmmsearch.exe'
Edit
argv = ' --tblout outfile hmmprofile fastafile' # your arguments
program = [r'"D:\\Python_Scripts\\HMMer3\\hmmsearch.exe"', argv]
subprocess.call(program)
I have a problem when programming in Python running under Windows. I need to work with file paths, that are longer than 256 or whatsathelimit characters.
Now, I've read basically about two solutions:
Use GetShortPathName from kernel32.dll and access the file in this way.
That is nice, but I cannot use it, since I need to use the paths in a way
shutil.rmtree(short_path)
where the short_path is a really short path (something like D:\tools\Eclipse) and the long paths appear in the directory itself (damn Eclipse plugins).
Prepend "\\\\?\\" to the path
I haven't managed to make this work in any way. The attempt to do anything this way always result in error WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: <path here>
So my question is: How do I make the 2nd option work? I stress that I need to use it the same way as in the example in option #1.
OR
Is there any other way?
EDIT: I need the solution to work in Python 2.7
EDIT2: The question Python long filename support broken in Windows does give the answer with the 'magic prefix' and I stated that I know it in this question. The thing I do not know is HOW do I use it. I've tried to prepend that to the path but it just failed, as I've written above.
Well it seems that, as always, I've found the answer to what's been bugging me for a week twenty minutes after I seriously ask somebody about it.
So I've found that I need to make sure two things are done correctly:
The path can contain only backslashes, no forward slashes.
If I want to do something like list a directory, I need to end the path with a backslash, otherwise Python will append /*.* to it, which is a forward slash, which is bad.
Hope at least someone will find this useful.
Let me just simplify this for anyone looking for a straight answer:
For python < 3: Path needs to be unicode, prepend string with u like u'C:\\path\\to\\file'
Path needs to start with \\\\?\\ (which is escaped into \\?\) like u'\\\\?\\C:\\path\\to\\file'
No forward slashes only backslashes: / --> \\
It has to be an absolute path; it does not work for relative paths
py 3.8.2
# Fix long path access:
import ntpath
ntpath.realpath = ntpath.abspath
# Fix long path access.
In my case, this solved the problem of running a script from a long path.
(https://developers.google.com/drive/api/v3/quickstart/python)
But this is not a universal fix.
It looks like the ntpath.realpath implementation has problems. This code replaced it with a dummy.
it works for me
import os
str1=r"C:\Users\manual\demodfadsfljdskfjslkdsjfklaj\inner-2djfklsdfjsdklfj\inner3fadsfksdfjdklsfjksdgjl\inner4dfhasdjfhsdjfskfklsjdkjfleioreirueewdsfksdmv\anotherInnerfolder4aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\5qbbbbbbbbbbbccccccccccccccccccccccccsssssssssssssssss\tmp.txt"
print(len(str1)) #346
path = os.path.abspath(str1)
if path.startswith(u"\\\\"):
path=u"\\\\?\\UNC\\"+path[2:]
else:
path=u"\\\\?\\"+path
with open(path,"r+") as f:
print(f.readline())
if you get a long path(more then 258 char) issue in windows then try this .
I'm trying to unzip a file from an FTP site. I've tried it using 7z in a subprocess as well as using 7z in the older os.system format. I get closest however when I'm using the zipfile module in python so I've decided to stick with that. No matter how I edit this I seem to get one of two errors so here are both of them so y'all can see where I'm banging my head against the wall:
z = zipfile.ZipFile(r"\\svr-dc\ftp site\%s\daily\data1.zip" % item)
z.extractall()
NotImplementedError: compression type 6 (implode)
(I think this one is totally wrong, but figured I'd include.)
I seem to get the closest with the following:
z = zipfile.ZipFile(r"\\svr-dc\ftp site\%s\daily\data1.zip" % item)
z.extractall(r"\\svr-dc\ftp site\%s\daily\data1.zip" % item)
IOError: [Errno 2] No such file or directory: '\\\\svr-dc...'
The catch with this is that it is actually giving me the first file name in the zip. I can see the file AJ07242013.PRN at the end of the error so I feel closer because it's at least getting to the point of reading the contents of the zip file.
Pretty much any iteration of this that I try gets me one of those two errors, or a syntax error but that's easily addressed and not my primary concern.
Sorry for being so long winded. I'd love to get this working, so let me know what you think I need to do.
EDIT:
So 7z has finally been added to the path and is running through without any errors with both the subprocess as well as os.system. However, I still can't seem to get anything to unpack. It looks to me, from all I've read in the python documentation that I should be using the subprocess.communicate() module to extract this file but it just won't unpack. When I use os.system it keeps telling me that it cannot find the archive.
import subprocess
cmd = ['7z', 'e']
sp = subprocess.Popen(cmd, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
sp.communicate('r"\C:\Users\boster\Desktop\Data1.zip"')
I don't think that sp.communicate is right but if I add anything else to it I have too many arguments.
python's zipfile doesn't support compression type 6 (imploded) so its simply not going to work. In the first case, that's obvious from the error. In the second case, things are worse. The parameter for extractfile is an alternate unzip directory. Since you gave it the name of your zip file, a directory of the same name can't be found and zipfile gives up before getting to the not-supported problem.
Make sure you can do this with 7z on the command line, try implementing subprocess again and ask for help on that technique if you need it.
Here's a script that will look for 7z in the usual places:
import os
import sys
import subprocess
from glob import glob
print 'python version:', sys.version
subprocess.call('ver', shell=True)
print
if os.path.exists(r'C:\Program Files\7-Zip'):
print 'have standard 7z install'
if '7-zip' in os.environ['PATH'].lower():
print '...and its in the path'
else:
print '...but its not in the path'
print
print 'find in path...'
found = 0
for p in os.environ['PATH'].split(os.path.pathsep):
candidate = os.path.join(p, '7z.*')
for fn in glob(candidate):
print ' found', fn
found += 1
print
if found:
print '7z located, attempt run'
subprocess.call(['7z'])
else:
print '7z not found'
Accoring to the ZipFile documentation, you might be better off copying the zip first to your working directory. (http://docs.python.org/2/library/zipfile#zipfile.ZipFile.extract)
If you have problems copying, you might want to store the zip in a path with no spaces or protect your code against spaces by using os.path.
I made a small test in which I used os.path.abspath to make sure I had the proper path to my zip and it worked properly.
Also make sure that for extractall the path that you specify is the path where the zip content will be extracted. (If a folder that is specified is not created, it will be created automatically) Your files will be extracted in your current working directory (CWD) if no parameter is passed to extractall.
Cheers!
Managed to get this to work without using the PIPE functionality as subprocess.communicate wouldn't unpack the files. Here was the solution using subprocess.call. Hope this can help someone in the future.
def extract_data_one():
for item in sites:
os.chdir(r"\\svr-dc\ftp site\%s\Daily" % item)
subprocess.call(['7z', 'e', 'data1.zip', '*.*'])
K.. I'm just using a simple script I found here:
import os
from os import rename, listdir
print os.listdir(".")
for filename in os.listdir("."):
if filename.startswith("colon-"):
print filename
os.rename(filename, filename[7:])
I need to basically take all files like colon-21.mp3 converted to 21.mp3.
But I get the error CANNOT CREATE A FILE WHEN THAT FILE ALREADY EXISTS.
How does one solve this? I am using Windows 7.
The problem is right here:
os.rename(filename, filename[7:])
Python indices start at 0, and the string "colon-" is only 6 characters long, so colon-21.mp3 will become 1.mp3 using your code. Change that line to use filename[6:] instead and your problem should be gone.
That said, using a hard coded string length like you are doing is not a good idea. It is error prone for exactly the reasons we have discovered here (hard coded numbers like this are often called "magic numbers" because it is difficult to tell why they are set to a given length). A superior alternative would be the following:
os.rename(filename, filename.split('-')[1])
I have written a few lines of code in Python to see if I can make it read a text file, make a list out of it where the lines are lists themselves, and then turn everything back into a string and write it as output on a different file. This may sound silly, but the idea is to shuffle the items once they are listed, and I need to make sure I can do the reading and writing correctly first. This is the code:
import csv,StringIO
datalist = open('tmp/lista.txt', 'r')
leyendo = datalist.read()
separando = csv.reader(StringIO.StringIO(leyendo), delimiter = '\t')
macrolist = list(separando)
almosthere = ('\t'.join(i) for i in macrolist)
justonemore = list(almosthere)
arewedoneyet = '\n'.join(justonemore)
with open('tmp/randolista.txt', 'w') as newdoc:
newdoc.write(arewedoneyet)
newdoc.close()
datalist.close()
This seems to work just fine when I run it line by line on the interpreter, but when I save it as a separate Python script and run it (myscript.py) nothing happens. The output file is not even created. After having a look at similar issues raised here, I have introduced the 'with' parameter (before I opened the output file through output = open()), I have tried flushing as well as closing the file... Nothing seems to work. The standalone script does not seem to do much, but the code can't be too wrong if it works on the interpreter, right?
Thanks in advance!
P.S.: I'm new to Python and fairly new to programming, so I apologise if this is due to a shallow understanding of a basic issue.
Where are the input file and where do you want to save the output file. For this kind of scripts i think that it's better use absolute paths
Use:
open('/tmp/lista.txt', 'r')
instead of:
open('tmp/lista.txt', 'r')
I think that the error can be related to this
It may have something to do with where you start your interpreter.
Try use a absolute path /tmp/randolista.txt instead of relative path tmp/randolista.txt to isolate the problem.