I have a batch script that uses xcopy to copy a large number of files. For a while, the following setup worked fine:
set source=<path_to_source_files>
set dest=<path_to_destination>
xcopy /y /s /i /r "%source%" "%dest%"
Today, the same command started producing an error:
File creation error - The system cannot find the path specified.
Furthermore, %errorlevel% was returning 4 (Initialization error occurred. There is not enough memory or disk space, or you entered an invalid drive name or invalid syntax on the command line).
I ruled out space issues, yet the other two possibilities didn't make sense given that many (most?) of the files had already been copied. Just in case, I tried using the /j switch to avoid buffering, but that didn't help.
In an attempt to gather more debugging info, I tried the /l switch to list the files being copied to see if it was any different from what xcopy was already displaying. For some reason, this actually resolved the issue! It even worked when I used the /q switch too (i.e. suppressing the xcopy messages):
xcopy /y /s /i /r /q /l "%source%" "%dest%"
According to this post, xcopy was deprecated back in 2011, so if this is a bug, Microsoft won't fix it. This post doesn't seem to apply either, so I have no idea why this is suddenly an issue (or why the /l switch resolves it, even when the list is then suppressed).
Primarily, this post is just to share the workaround, but hopefully someone can explain the underlying issue and/or workaround.
Updates: the previous workaround doesn't seem to actually work as expected; apparently, with the /l switch, xcopy can finish with %errorlevel%=0 without actually copying anything.
Looking more closely at the list of files, the problematic ones come from Python installation files - specifically, ones from the tensorflow package.
I typically run the script on a VMware VM, but even on my local machine, I consistently hit the same issue on the same file when I omit the /l switch. Interestingly, the problematic files on my local machine are slightly different than the problematic files on a VM (i.e. my local machine progresses further through the tensorflow folder than a VM).
Sometimes, adding the /h switch does the trick, but this doesn't work consistently. The most reliable method seems to be the use of robocopy instead xcopy, but even that command reports a few skipped files.
Originally, the files are zipped to a .tar.gz file, and I sequentially extract the files using 7zip before copying them. I've tried extracting the files directly to the desired directory directly, but the -spe switch doesn't work as expected either.
All things considered, I'm starting to suspect a limitation with tensorflow files.
Glad to know that this issue resolved and thanks for confirming on the actual issue with its resolution.
"The paths didn't seem invalid at first, but then I noticed that the
output used relative paths, not full paths. The unzipped folder name
was unnecessarily long, and renaming it helped."
Hi #Mofi, Thanks for the helpful comment and posting this answer in this answer section for the community benefit.
Difficult to answer the question without knowing which files with their fully qualified file names could not be copied
into which destination directory. It could be that the MAX_PATH
limitation (259 characters + a string terminating null character = 260
characters) is responsible for the error message. The listing could
work on fully qualified source file name that has less than 260
characters while the fully qualified destination file name has 260 or
more characters. There are other file system limitations as well
Related
I have a short script written in python that saves a simple .txt to the same directory the .py file is in. Problem is: It only does so on one of my two computers.
My code doesn't include a hard-coded path to write to. On my laptop, I can put the makemeanote.py in any folder and it will create the note right there. On my desktop pc, all the notes end up in System32. PATH is set exactly the same way on both machines, and both use Windows\py.exe as the executable.
Somewhat interestingly, I only get an admin-screen on the desktop pc, asking if I want to allow changes to my system by "Built: Release_master_v3.8.2_"etc., whereas on my laptop it simply runs and does its job.
No amount of un- and reinstalling has changed anything, even when I thought I had eradicated any trace of python on my hard drive. Both PCs use current Win10/64 installations.
What is happening there?
A clarification: It's not about fixing the bug, it's about understanding the inconsistent behaviour! I know I could just hard-code any directory, but that takes away the beauty, don't you think?
I finally found the cause of the weird behavior:
For some reason, the py.exe in my WINDOWS folder was set to always be run as admin. As soon as I unchecked that option, the User Account Control (UAC) check disappeared and my file finally behaved as I had expected.
I am building an application that needs access to a resource in the same folder (It needs the xsd file to validate the input file).
How can I reach this xsd file reliably?
To elaborate:
I need to be able to call the executable:
cd not/the/directory/of/the/program
path/to/program -c config.xml
What is the path to use here?
subprocess.Popen("xmllint --noout --schema ?/?/config.xsd " + configfilename)
Take a look at the zeroth command-line argument your program gets. It's usually the name of the program, and might also include a path to the program. Combine that with the current directory. Then use realpath or some equivalent of it to get rid of symbolic links. However, I think there are still some corner cases where this approach might fail.
You didn't specify /the programming language/ or operating system you are using. If you're writing a bash script, see here:
https://stackoverflow.com/a/246128/11451509
For Python, see How do I get the path of the current executed file in Python?
If the operating system is Linux, you could also go through the /proc pseudo filesystem, as explained here:
https://stackoverflow.com/a/738024/11451509
I am using pyDev with eclipse.
I import external code to my project, and I got bad indentation warning , that's because, the external code has different indentation than my code.
How could I fix that ?
"Bad indentaion" warning could be turned off with Preferences -> PyDev -> Editor -> Code Analysis -> Others -> "indentation problems and mixing of tabs and spaces".
Once the value is set to "ignore". Those markers should go away. :)
There is a script installed in %PYTHON%/Tools/Scripts folder called reindent.py. I haven't used it myself, but according to the documentation it fixes the indentation of your code to 4 spaces:
reindent [-d][-r][-v] [ path ... ]
-d (--dryrun) Dry run. Analyze, but don't make any changes to, files.
-r (--recurse) Recurse. Search for all .py files in subdirectories too.
-n (--nobackup) No backup. Does not make a ".bak" file before reindenting.
-v (--verbose) Verbose. Print informative msgs; else no output.
-h (--help) Help. Print this usage information and exit.
Change Python (.py) files to use 4-space indents and no hard tab characters.
Also trim excess spaces and tabs from ends of lines, and remove empty lines
at the end of files. Also ensure the last line ends with a newline.
If no paths are given on the command line, reindent operates as a filter,
reading a single source file from standard input and writing the transformed
source to standard output. In this case, the -d, -r and -v flags are
ignored.
You can pass one or more file and/or directory paths. When a directory
path, all .py files within the directory will be examined, and, if the -r
option is given, likewise recursively for subdirectories.
If output is not to standard output, reindent overwrites files in place,
renaming the originals with a .bak extension. If it finds nothing to
change, the file is left alone. If reindent does change a file, the changed
file is a fixed-point for future runs (i.e., running reindent on the
resulting .py file won't change it again).
The hard part of reindenting is figuring out what to do with comment
lines. So long as the input files get a clean bill of health from
tabnanny.py, reindent should do a good job.
The backup file is a copy of the one that is being reindented. The ".bak"
file is generated with shutil.copy(), but some corner cases regarding
user/group and permissions could leave the backup file more readable than
you'd prefer. You can always use the --nobackup option to prevent this.
In a Python source distribution it is located in the Tools/scripts directory, and in Ubuntu you can install the python2.7-examples or python3.4-examples packages.
I was trying to run Python 3.3 off of my flash drive. I already tried Portable Python, but most of what it had wouldn't open, and it crashed my laptop.
So, I deleted it, and installed regular Python. Then, I wanted to start adding my favorite modules. And, I needed a way to start IDLE without the original shortcut.
To install modules, I added my Python install to my PATH variable.
To make all this easier, I made a batch file, using some code I found on this question.
So far, I have this. It also asks for the drive letter, because that changes from computer to computer.
#echo off
echo This only works if your Python install is added to the system-wide PATH variable
set /p Path="Enter the Drive Letter on this computer. No Symbols, Just the Letter, Capital"
cd %Path%:\Program Files\Python33
echo type this when python comes up...
echo import idlelib.PyShell
echo idlelib.PyShell.main()
echo.
echo.
echo.
echo.
python
It outputs this:
If you go on and follow the instructions and type what it says, it brings up IDLE.
I couldn't figure out how to get the batch file to actually type into the Python prompt, so I told it to tell the user to type what needed to be typed.
What I need to know is, how can I change the PATH variable from within the batch file. Also, how to I remove it when I'm done (this isn't as important, and could even be in a separate batch file).
Or, alternatively, is there a way just to shortcut to IDLE?
Also, is there a way to run .py files without the command line, with the Python install on my flash drive?
Thanks!
You can invoke Python with this command line:
python -c"import idlelib.PyShell;idlelib.PyShell.main()"
and it will bring-up IDLE shell without the need for the user to type anything.
Edit: BTW, Are you sure you really need to change the global path settings. Try to see if the following script can start Python the way you want. You must put it in the root of the USB drive where you have your Python installation.
#echo off
setlocal
set SCRIPT_DIR=%~dp0
:: Removes trailing backslash (for readability in the following)
set SCRIPT_DIR=%SCRIPT_DIR:~0,-1%
set PYTHON_HOME=%SCRIPT_DIR%\Program Files\Python33
set PATH=%PYTHON_HOME%;%PATH%
"%PYTHON_HOME%\python.exe" -c"import idlelib.PyShell;idlelib.PyShell.main()"
Edit: Every process has an associated environment, which is a set of name-value pairs called environment variables. When a process is started it gets a copy of the environment of its parent process. The global OS settings for environment variables are used for processes started directly from the OS (GUI or command line) shell. The set command in batch files sets or modifies an environment variable in the environment of the current process (not globally).
All the set commands you see in the above script change only the environment of the current process. These changes will be seen by the process created by the last line (python.exe) because it is a child process of the command shell (cmd.exe) process that is executing the batch-file.
The line
set PATH=%PYTHON_HOME%;%PATH%
prepends the content of the PYTHON_HOME variable to the PATH variable of the current process. For example, if PATH were c:\foo\bar;d:\aaa\bbb and PYTHON_HOME were c:\python then the new value of PATH will be c:\python;c:\foo\bar;d:\aaa\bbb
It can not be guaranteed this is possible unless you have high enough system privileges that you can change the global path. There is really no way around this on most computers that you do not own, which I imagine is the main purpose. In those cases when you have enough privileges (it is worth a try some systems still allow this for regular users but many others do not) you can use:
setx PATH "%path%;yourpath"
edit and ps:
You can figure out the drive letter without input, if you know the disk label, with something like this:
#echo off
set label=DRIVENAME
set cmd=WMIC logicaldisk WHERE volumename^^="%label%" GET caption
FOR /F "tokens=1" %%G IN ('%cmd% ^| find ":"')DO set pydrive=%%G
echo %pydrive%\pathtopython
rem do your stuff here
the idle started inside the batch will inherit path but other instances will not. Hard to test conclusively tough.
Explanation of the batch script above. The command wmic is short for windows management instrumentation commandline. One can use WMI to do many things one of them is to issue WQL (SQL for WMI) queries as if windows would be a a database. Databases contain many tables in this case the computer is instructed to fetch the table named logicaldisk. Table logicaldisk has 38 columns and one row for each disk connected to the system. This is way to much data for this purpose. So the data is filtered. WHERE causes the database only to spit out rows that contain some specific value in this case its only interested in rows where column volumename ins equal to DRIVENAME, likewise you could use serial number size or any other criteria. Lastly GET is used to limit the columns you get back as results since your only interested in the letter name of the drive that's what you ask. That is called a caption in the table so that what you ask.
Since the command is a bit long, so i put the command inside a variable (not the result) this shortens the for line so it fits stack overflow. Since = need to be escaped i need to use the escape sequence ^ too times so it still es capable in the for loop.
The for loop is used to capture the return value of the wmic command. Since the answer has many lines i filter for only the lines which contain a colon character. And put it to variable pydrive.
I've been following https://github.com/kripken/emscripten/wiki/Using-Emscripten-on-Windows along up to now. When I run the test suite in tests/runner.py, some tests fail with a python error.
subprocess.Popen is throwing error 2 on files that clearly either exist or should be getting created mid-script (the latter being, ie, files in the C:/tmp directory, which I have created and also made certain is not read-only).
I have no idea what's going wrong or how I can fix this.
I set an environment variable EM_SAVE_DIR=1 as recommended on the github page.... Though it was recommended for debugging purposes, the fact of the matter is that it caused the files to remain both in memory and in the correct location, such that the error now only gets thrown on executables that I mistakenly didn't have anywhere on my PATH (cmake, sh, and make).
Though at some point later today, if I get all these tests worked out for runner.py, I may need to ask about making it work on visual studio, which throws error 6 on not finding a correct file in the tmp directory, and seems to be ignoring EM_SAVE_DIR=1....