Develop Reference

Shutil.move inside a recursive function

I wrote this function to achieve a similar behavior as in Windows when moving files and directories. In particular, objects should be overwritten.
from pathlib import Path
import shutil
import os
def move_anyhow(source: Path | str, dest: Path | str) -> Path:
"""
Move source (directory or file) and overwrite files with same name in dest if exists.
"""
try:
shutil.move(source, dest)
except shutil.Error:
if source.is_file():
shutil.move(source, dest / source.name)
else:
for path in source.iterdir():
move_anyhow(path, dest / source.name)
os.rmdir(source)
return dest / source.name
I took a recursive approach to moving nested source directories like this one
.../source/
dir_A/
dir_B/
file_X
to destination
.../dest/
dir_A/
dir_B/
file_X
file_Y
On production I get a PermissionError now and then which looks like this:
PermissionError: [Errno 13] Permission denied: '/delivery/post/01_FROM_CF/W22_FW/50479944_003' -> '/delivery/post/01_FROM_CF/ERROR/W22_FW/50479944_003'
File "shutil.py", line 813, in move
os.rename(src, real_dst)
OSError: [Errno 39] Directory not empty: '/delivery/post/01_FROM_CF/W22_FW/50479944_003'
File "ors/path.py", line 34, in move_anyhow
shutil.move(source, dest)
File "shutil.py", line 831, in move
rmtree(src)
File "shutil.py", line 728, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "shutil.py", line 726, in rmtree
os.rmdir(path)
All files were moved but the empty source folder remained. I can't reproduce this error locally. So I my first guess was that this is a server issue. Still, I wonder if the nested approach could cause this error.
So I guess my question is whether an catched shutil.move error can block another shutil.move operation of a file inside the source directory.

The problem will probably be in the timing of files deletion and directory deletion. I would try to cancel the immediate deletion of directories and I would delete the directories only at the end.
Remove the command line:
os.rmdir(source)
and when the file transfer is complete, then call the command
shutil.rmtree(source)
Good luck.

Subprocess.run() cannot find file, even if path.exists() returns true

I found these two pages:
Subprocess.run() cannot find path
Python3 Subprocess.run cannot find relative referenced file
but it didn't help. The first page talks about using \\ but I already do, and the second one talks about double quotes around one of the arguments.
work = Path("R:\\Work")
resume = work.joinpath("cover_letter_resume_base.doc")
current_date = construct_current_date()
company_name = gather_user_information(question="Company name: ",
error_message="Company name cannot be empty")
position = gather_user_information(question="Position: ",
error_message="Position cannot be empty")
# Construct destination folder string using the company name, job title, and current date
destination = work.joinpath(company_name).joinpath(position).joinpath(current_date)
# Create the destintion folder
os.makedirs(destination, exist_ok=True)
# Construct file name
company_name_position = "{0}_{1}{2}".format(company_name.strip().lower().replace(" ", "_"),
position.strip().lower().replace(" ", "_"), resume.suffix)
resume_with_company_name_job_title = resume.stem.replace("base", company_name_position)
destination_file = destination.joinpath(resume_with_company_name_job_title)
# Copy and rename the resume based on the company and title.
shutil.copy2(src=resume, dst=destination_file)
if destination_file.exists():
print(f"{destination_file} created.")
#subprocess.run(["open", str(destination_file)], check=True)
The program gets the company name and position from the user, generates the current date, creates the directories, and then moves/renames the base resume based on the user input.
Output and Results:
Company name: Microsoft
Position: Software Eng
R:\Work\Microsoft\Software Engineer\20190722\cover_letter_resume_microsoft_software_eng.doc
created.
Error Message:
[WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File "c:/Users/Kiska/python/job-application/main.py", line 59, in <module>
main()
File "c:/Users/Kiska/python/job-application/main.py", line 53, in main
raise error
File "c:/Users/Kiska/python/job-application/main.py", line 48, in main
subprocess.run(["start", str(destination_file)], check=True)
File "C:\Program Files (x86)\Python37-32\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Program Files (x86)\Python37-32\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\Program Files (x86)\Python37-32\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
The if statement returns True but subprocess.run() cannot see the file, but I'm not really sure why.

On which operating system are you? The backslashes in your path suggest that you're on Windows and you're using open to open the document with its default application. However, looking at this question Open document with default OS application in Python, both in Windows and Mac OS you should use start instead of open for Windows:
subprocess.run(["start", str(destination_file)], check=True, shell=True)
Also you need to add shell=True for start to work. However, you should read https://docs.python.org/3/library/subprocess.html#security-considerations beforehand.
(I suspect, the error [WinError 2] The system cannot find the file specified appears, because Windows cannot find open - it's not about the document you're trying to open.)

Python. Send Uploaded File To Remote Server

In My Flask App, i want to upload a file to a remote server.
i tried this code but i get an error
import subprocess
import os
c_dir = os.path.dirname(os.path.abspath(__file__))
myfile = open(c_dir + '\\cape-kid.png')
p = subprocess.Popen(["scp", myfile, destination])
sts = os.waitpid(p.pid, 0)
this was just a test file. there's an image in the same directory as my test python file. the error said:
Traceback (most recent call last): File
"C:\Users\waite-ryan-m\Desktop\remote-saving\test-send.py", line 20,
in
p = subprocess.Popen(["scp", c_dir + '\cape-kid.png', 'destination']) File
"C:\Users\waite-ryan-m\Desktop\WPython\WinPython-64bit-2.7.12.1Zero\python-2.7.12.amd64\lib\subprocess.py",
line 711, in init
errread, errwrite) File "C:\Users\waite-ryan-m\Desktop\WPython\WinPython-64bit-2.7.12.1Zero\python-2.7.12.amd64\lib\subprocess.py",
line 959, in _execute_child
startupinfo) WindowsError: [Error 2] The system cannot find the file specified

With open() you open an file to read or write on it. What you want is to concatinate the string and use this as parameter for scp. Maybe the file you want to copy also doesn't exist - have you tried printing the path you constructed and checking it manually?
And have you defined destination anywhere? This message could also mean, that the system cannot find scp.

Executing a vbs file with arguments created by python

I would like to convert dozens of excel sheets to csv files at once. I have a working .vbs file which makes the conversion, and I would like to execute this .vbs file on the different sheets with the help of a python code. I have the following 2 versions of the python code:
Version 1:
import os
import sys
import subprocess
FolderName=sys.argv[1]
FileList=os.listdir(FolderName)
NewList=[]
for i in FileList:
NewItem=i.split('.xls')
NewXls=FolderName+"\\"+NewItem[0]+".xlsx "
NewCsv=FolderName+"\\"+NewItem[0]+".csv"
NewCommand="C:\\Users\\user\\XlsToCsv.vbs "+sys.argv[2]+" "+NewXls+NewCsv
subprocess.call(NewCommand)
Version 2:
import os
import sys
import subprocess
def main(directory,extension,sheet):
for filename in os.listdir(directory):
if filename.endswith(extension):
path = os.path.join(directory, filename)
base = os.path.join(directory, filename[:len(filename)-len(extension)])
print base
new_xls = base + extension
new_csv = base + '.csv'
subprocess.call(['C:\\Users\\user\\XlsToCsv.vbs', sheet, new_xls, new_csv])
main(sys.argv[1],sys.argv[2],sys.argv[3])
It does not matter, which I try, I get the same error message:
Traceback (most recent call last):
File "C:/Users/user/Desktop/Work/XlsDir.py", line 16, in <module>
subprocess.call(NewCommand)
File "C:\Python27\lib\subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python27\lib\subprocess.py", line 711, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 948, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 er ikke et gyldigt Win32-program
The last line of the error message means approximately, that it is not a valid Win32-program.
What I have tried so far:
If I run the .vbs file from command prompt with the right arguments (sheet, name of the .xls file and name of the .csv file) then it works fine.
If I print the commands that python generates and copy them into command prompt, they work fine.
I tried every combinations of '\' and '\' within the different paths, and nothing got any better.
I tried to execute the programs with replacing the sys.argv[i] arguments with specific arguments and then execute the .py file from command prompt. I get the same error message.
I hope some of you can help me. Thanks a lot!

To elaborate on Ansgar's remedy:
Starting a .vbs from the command line 'works', because the shell associates the extension .vbs with an application (e.g. cscript/wscript; see ftype, assoc, cscript //E, cescript //S).
subprocess.call() does not open a shell, so either specify the application (c|wscript.exe) or start the shell yourself:
import subprocess
#subprocess.call("notepad") # works
#subprocess.call("dir") # [Error 2] The system cannot find the file specified
# no shell, no intrinsics
#subprocess.call("19112944.vbs") # [Error 193] %1 is not a valid Win32 application
# no shell, can't associate .vbs with c|wscript.exe
subprocess.call("cscript 19112944.vbs") # works
subprocess.call("cmd /c 19112944.vbs") # works
# have shell, can associate .vbs with c|wscript.exe

Try running the script with cscript.exe:
subprocess.call(['cscript.exe', 'C:\\Users\\user\\XlsToCsv.vbs', sheet, new_xls, new_csv])

How to use the dir/s command in Python?

Background
I use the command dir/s in batch files all the time. But, I am unable to call this using python. NOTE: I am using Python 2.7.3.
Code
import subprocess
subprocess.call(["dir/s"])
Error Message
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
subprocess.call(["dir/s"])
File "C:\Python27\lib\subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python27\lib\subprocess.py", line 679, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 896, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
I have tried changing the quotations but nothing has worked.
How would I call the dir/s module using subprocess?

How about
subprocess.call("dir/s", shell=True)
Not verified.

This is a lot different than what you're asking but it solves the same problem. Additionally, it solves it in a pythonic, multiplatform way:
import fnmatch
import os
def recglob(directory, ext):
l = []
for root, dirnames, filenames in os.walk(directory):
for filename in fnmatch.filter(filenames, ext):
l.append(os.path.join(root, filename))
return l

You need a space between dir and /s. So break it into an array of 2 elements. Also as carlosdoc pointed out, you would need to add shell=True, since the dir command is a shell builtin.
import subprocess
subprocess.call(["dir", "/s"], shell=True)
But if you're trying to get a directory listing, make it OS independent by using the functions available in the os module such as os.listdir(), os.chdir()

I finally found the answer. To list all directories in a directory (e.g. D:\\, C:\\) on needs to first import the os module.
import os
Then, they need to say that they want to list everything. Within that, they need to make sure that the output is printed.
for top, dirs, files in os.walk('D:\\'):
for nm in files:
print os.path.join(top, nm)
That was how I was able to solve it. Thanks to this.

As it's an inbuilt part of the command line you need to run it as:
import subprocess
subprocess.call("cmd /c dir /s")