Develop Reference

why does the list switch allow xcopy to work?

I have a batch script that uses xcopy to copy a large number of files. For a while, the following setup worked fine:
set source=<path_to_source_files>
set dest=<path_to_destination>
xcopy /y /s /i /r "%source%" "%dest%"
Today, the same command started producing an error:
File creation error - The system cannot find the path specified.
Furthermore, %errorlevel% was returning 4 (Initialization error occurred. There is not enough memory or disk space, or you entered an invalid drive name or invalid syntax on the command line).
I ruled out space issues, yet the other two possibilities didn't make sense given that many (most?) of the files had already been copied.  Just in case, I tried using the /j switch to avoid buffering, but that didn't help.
In an attempt to gather more debugging info, I tried the /l switch to list the files being copied to see if it was any different from what xcopy was already displaying. For some reason, this actually resolved the issue! It even worked when I used the /q switch too (i.e. suppressing the xcopy messages):
xcopy /y /s /i /r /q /l "%source%" "%dest%"
According to this post, xcopy was deprecated back in 2011, so if this is a bug, Microsoft won't fix it. This post doesn't seem to apply either, so I have no idea why this is suddenly an issue (or why the /l switch resolves it, even when the list is then suppressed).
Primarily, this post is just to share the workaround, but hopefully someone can explain the underlying issue and/or workaround.
Updates: the previous workaround doesn't seem to actually work as expected; apparently, with the /l switch, xcopy can finish with %errorlevel%=0 without actually copying anything.
Looking more closely at the list of files, the problematic ones come from Python installation files - specifically, ones from the tensorflow package.
I typically run the script on a VMware VM, but even on my local machine, I consistently hit the same issue on the same file when I omit the /l switch. Interestingly, the problematic files on my local machine are slightly different than the problematic files on a VM (i.e. my local machine progresses further through the tensorflow folder than a VM).
Sometimes, adding the /h switch does the trick, but this doesn't work consistently. The most reliable method seems to be the use of robocopy instead xcopy, but even that command reports a few skipped files.
Originally, the files are zipped to a .tar.gz file, and I sequentially extract the files using 7zip before copying them. I've tried extracting the files directly to the desired directory directly, but the -spe switch doesn't work as expected either.
All things considered, I'm starting to suspect a limitation with tensorflow files.

Glad to know that this issue resolved and thanks for confirming on the actual issue with its resolution.
"The paths didn't seem invalid at first, but then I noticed that the
output used relative paths, not full paths. The unzipped folder name
was unnecessarily long, and renaming it helped."
Hi #Mofi, Thanks for the helpful comment and posting this answer in this answer section for the community benefit.
Difficult to answer the question without knowing which files with their fully qualified file names could not be copied
into which destination directory. It could be that the MAX_PATH
limitation (259 characters + a string terminating null character = 260
characters) is responsible for the error message. The listing could
work on fully qualified source file name that has less than 260
characters while the fully qualified destination file name has 260 or
more characters. There are other file system limitations as well

I'm trying to set up python in notepad++ and it can't understand my filepath that has a space in it

I'm using this tutorial: https://silentcrash.com/2016/12/run-python-script-notepad/
The example for Python 3 includes the folder "Program Files (x86)", which I assume is not necessary, but I would still like to know how to fix this problem if it comes up in the future and I don't have another way of doing it.
So what happens is that the program assumes the space is the end of the command and it move on to the argument as seen here:
The Program to Run
Error
The system cannot find the file specified.
An attempt was made to execute the below command.
------------------------------------------------
Command: C:\Program
Arguments: Files\Python37\Lib\idlelib.py "new 2"
Error Code: 2
Is there any way to write the file path such that it won't be split up like this?

You need to surround your arguments with quotes when they contain spaces. In your example, you would use the following command in the Run... dialog:
"C:\Program Files\Python37\Lib\idlelib.py" "$(FULL_CURRENT_PATH)"
I found a helpful hint on superuser.com that will stop the console window from closing when your script finishes. However, note that it now needs an extra set of quotes.
cmd /k ""C:\Program Files\Python37\Lib\idlelib.py" "$(FULL_CURRENT_PATH)""

Applying 2to3 in PyDev project in Eclipse does not affect any changes to files

I opened a Python 2 project in PyDev on Eclipse with a Python3 interpreter installed, so I get errors in certain places.
The project's context menu has a PyDev -> "Apply 2to3" option. When I select it, I see diff output in my console but the .py files in the project are not changed. I don't see any obvious way to apply the diff output to make those changes.
Shouldn't "Apply 2to3" perform changes to the files and leave the project ready to compile under Python 3?

You have to enter the parameters for 2to3 to do that...
I just saw that the dialog is currently horrible (it doesn't show the full text -- back then the text from 2to3 was probably small, but now that it's big it doesn't fit on the dialog space very well and the content is cropped to what's visible -- just fixed it: https://github.com/fabioz/Pydev/commit/fb0aa9d28c15e7deeb3aa229b0ad66146802d9e4).
Now, to answer your question, at the dialog that opens you need to pass -w to write the files creating a backup file or -w -n if you want to write the files but don't want to keep backup files -- which I recommend if your source code is already under source control ;)

Python script gives `: No such file or directory`

I have several python scripts which work just fine but one script has (as of this morning) started giving me this error if I try to run it from the bash:
: No such file or directory
I am able to run the 'broken' script by doing python script_name.py and after looking around a bit the general idea that I picked up was that maybe my line ending of the hashbang got changed (silently) so I looked at the line ending of a working script and a broken script via the :set list option in VI as indicated in this question -> View line-endings in a text file
Both files appear to end using the same character (a $) so I am kind of at a loss on how to proceed from here. Specifically, how to actually 'see' the line ending in case the set list was not the right method.
PS: The script is executable and the shebang is in there, I stated that it's just this 1 script that was working fine before the weekend but it started giving me this error as of this morning.
-- edit: --
Running the script through dos2unix does get it working again but I would like to know of any way to visualize the line ending somehow in VI(M) or why Geany somehow converted the line endings in the first place (as I never work on a dos/windows system anyhow).

From the comments above it looks like you have dos line endings, and so the hashbang line is not properly processed.
Line ending style are not shown with :set list in Vim because that option is only used when reading/writing the file. In memory line endings are always that, line-endings. The line ending style used for a file is kept in a Vim per-file option, weirdly called fileformat.
To see/change the line ending style from Vim, you can use the following commands:
:set fileformat
:set ff
It will show dos or unix. You want unix, of course ;-).
To change it quickly you can save the file with:
:w ++ff=unix
Or if you prefer:
:set ff=unix
And then save the file normally.
So see all the gory details just do :help fileformat, :help file-formats and :help fileformats

You can also use the dos2unix command to convert the file format
dos2unix
This helped me to run the python scripts
This normally happens when we open files in windows do changes and save it.
if you open the file locate the ^M characters at the end of every line
Thanks

Personally, I find it kinda wrong using direct path to python interpreter. As you dont use windows platform, you should have program env, usually in /usr/bin (/usr/bin/env). Try using following shebang:
#!/usr/bin/env python
Different distros store python binary in /bin or /usr/bin (or some weird locations), and this one makes your script config-independent (as far as possible, here we have possibility that env is stored elsewhere; still - it is less possible that env is not in /usr/bin than that python is mislocated).
I had similiar problem (if not exactly the same) and that worked for me.
Also, I have both python interpreters (2.7.x and 3.x) installed, so I need to use "python3" argument for env. AFAIR usually distros link different names to different binaries, so "env python" will run python2.7 on my system, "env python3" (also python33, or smth like that) will run p3k, and "env python2" (also python27, etc) will run python 2.7.x. Declaring which version of interpreter should be used seems like a good idea too.

I came across this problem editing my code on Windows, checking it in with git, and checking out and running it on Linux.
My solution was: tell git to Do The Right Thing. I issued this command on the Windows box:
git config --global core.autocrlf true
Modified the files and checked them in; voila, no such problem any more.
As discussed on the Git documentation.

Python off a Flash Drive, how to start IDLE?

I was trying to run Python 3.3 off of my flash drive. I already tried Portable Python, but most of what it had wouldn't open, and it crashed my laptop.
So, I deleted it, and installed regular Python. Then, I wanted to start adding my favorite modules. And, I needed a way to start IDLE without the original shortcut.
To install modules, I added my Python install to my PATH variable.
To make all this easier, I made a batch file, using some code I found on this question.
So far, I have this. It also asks for the drive letter, because that changes from computer to computer.
#echo off
echo This only works if your Python install is added to the system-wide PATH variable
set /p Path="Enter the Drive Letter on this computer. No Symbols, Just the Letter, Capital"
cd %Path%:\Program Files\Python33
echo type this when python comes up...
echo import idlelib.PyShell
echo idlelib.PyShell.main()
echo.
echo.
echo.
echo.
python
It outputs this:
If you go on and follow the instructions and type what it says, it brings up IDLE.
I couldn't figure out how to get the batch file to actually type into the Python prompt, so I told it to tell the user to type what needed to be typed.
What I need to know is, how can I change the PATH variable from within the batch file. Also, how to I remove it when I'm done (this isn't as important, and could even be in a separate batch file).
Or, alternatively, is there a way just to shortcut to IDLE?
Also, is there a way to run .py files without the command line, with the Python install on my flash drive?
Thanks!

You can invoke Python with this command line:
python -c"import idlelib.PyShell;idlelib.PyShell.main()"
and it will bring-up IDLE shell without the need for the user to type anything.
Edit: BTW, Are you sure you really need to change the global path settings. Try to see if the following script can start Python the way you want. You must put it in the root of the USB drive where you have your Python installation.
#echo off
setlocal
set SCRIPT_DIR=%~dp0
:: Removes trailing backslash (for readability in the following)
set SCRIPT_DIR=%SCRIPT_DIR:~0,-1%
set PYTHON_HOME=%SCRIPT_DIR%\Program Files\Python33
set PATH=%PYTHON_HOME%;%PATH%
"%PYTHON_HOME%\python.exe" -c"import idlelib.PyShell;idlelib.PyShell.main()"
Edit: Every process has an associated environment, which is a set of name-value pairs called environment variables. When a process is started it gets a copy of the environment of its parent process. The global OS settings for environment variables are used for processes started directly from the OS (GUI or command line) shell. The set command in batch files sets or modifies an environment variable in the environment of the current process (not globally).
All the set commands you see in the above script change only the environment of the current process. These changes will be seen by the process created by the last line (python.exe) because it is a child process of the command shell (cmd.exe) process that is executing the batch-file.
The line
set PATH=%PYTHON_HOME%;%PATH%
prepends the content of the PYTHON_HOME variable to the PATH variable of the current process. For example, if PATH were c:\foo\bar;d:\aaa\bbb and PYTHON_HOME were c:\python then the new value of PATH will be c:\python;c:\foo\bar;d:\aaa\bbb

It can not be guaranteed this is possible unless you have high enough system privileges that you can change the global path. There is really no way around this on most computers that you do not own, which I imagine is the main purpose. In those cases when you have enough privileges (it is worth a try some systems still allow this for regular users but many others do not) you can use:
setx PATH "%path%;yourpath"
edit and ps:
You can figure out the drive letter without input, if you know the disk label, with something like this:
#echo off
set label=DRIVENAME
set cmd=WMIC logicaldisk WHERE volumename^^="%label%" GET caption
FOR /F "tokens=1" %%G IN ('%cmd% ^| find ":"')DO set pydrive=%%G
echo %pydrive%\pathtopython
rem do your stuff here
the idle started inside the batch will inherit path but other instances will not. Hard to test conclusively tough.
Explanation of the batch script above. The command wmic is short for windows management instrumentation commandline. One can use WMI to do many things one of them is to issue WQL (SQL for WMI) queries as if windows would be a a database. Databases contain many tables in this case the computer is instructed to fetch the table named logicaldisk. Table logicaldisk has 38 columns and one row for each disk connected to the system. This is way to much data for this purpose. So the data is filtered. WHERE causes the database only to spit out rows that contain some specific value in this case its only interested in rows where column volumename ins equal to DRIVENAME, likewise you could use serial number size or any other criteria. Lastly GET is used to limit the columns you get back as results since your only interested in the letter name of the drive that's what you ask. That is called a caption in the table so that what you ask.
Since the command is a bit long, so i put the command inside a variable (not the result) this shortens the for line so it fits stack overflow. Since = need to be escaped i need to use the escape sequence ^ too times so it still es capable in the for loop.
The for loop is used to capture the return value of the wmic command. Since the answer has many lines i filter for only the lines which contain a colon character. And put it to variable pydrive.