Running two .py files in cmd via a .bat? - python

EDIT2: Stress tested the fix, it still failed after creating near 1TB of intermediary files. Changed .py code to delete intermediary files after performing necessary processes. Stress test succeeded after this change. Original issues likely to do with memory. Still no way of proving.
EDIT: I'm still not sure whether or not it was a memory issue. However I got the full process to run via .bat by breaking up the second script into 4 additional .py files and running those instead of the full second script at once. Problem solved for now.
ORIGINAL:
I am running two .py files through cmd via a .bat file. The reason I am doing this is the first .py is an arcpy file that requires use of 32 bit python, and the second .py is a combined PCI and arcpy file that requires use of 64 bit python.
The first file runs without issue. The second file gets to a certain point, the same point every time, and I am prompted with a "python.exe has stopped working" dialog box. This point is in the middle of a for loop in the PCI section of code. I have run the PCI python in PCI's interpreter without issue. I suspect this might be related to memory issues but am uncertain, and have no idea how to check.
How do I check to see if memory issues are the problem?
.bat code below:
C:/Python27/ArcGIS10.5/python.exe E:/Path/CODE/file1.py
C:/Python27/ArcGISx6410.5/python.exe E:/Path/CODE/file2.py

Related

Can't move a folder despite it not being used - Windows | Python

I have encountered an issue where I have a file in Onedrive that I am not able to move. I have determined that, despite it saying that the action cant be completed because the folder or file is open in another program, it is not being used by another program.
I have come to this conclusion by using resource monitor to check which apps are using the folder. There are none besides file explorer.
Context as it's probably important:
The reason why this has happened is that i deleted a python venv in the file directory, resulting in my VS Code interpreter breaking. I have restored this venv file but the issue still has not been fixed. According to this post, it is the result of workspace links, however, I have never once used workspaces. The general idea that a file WITHIN that unmovable directory could be the reason though, and as a result, is possibly why this issue may be occuring. What does not make sense is why it is still happening even without VS Code running or any service connections being made as evident through resource monitor
Another thing to note is that the folder has not completely synced, however, based on past experience, this should not be an issue.
I successfully moved the file via terminal, however the issue still remains about why i could not simply move the file with file explorer.

Running a python script and changing git branch

I am trying to find ways to make better use of my time while programming.
I have a python script that does some heavy work (it can take hours) to finish. Now, most of the work it does is network related, so i have plenty of cpu resources to spare.
If the script was a C binary executable, it would be fine to git checkout onto a different branch and do extra work, I could even modify the binary in disk as it has been copied to ram, so until it finishes running I won't affect program output.
But python scripts are translated, not compiled. What happens if I start tampering with the source file, can i corrupt the programs output, or is the text file and associated imports copied to RAM, allowing me to tamper with the source with no risk of changing the behaviour of the running program?
In general, if you have a single Python file which you run as a script, you're fine. When you run the file, it is compiled into bytecode which is then executed. You can change the original script at this point and nothing breaks.
However, we can deliberately break it by writing some horrible but legal code like this:
horrible.py:
from time import sleep
sleep(10)
import silly
silly.thing()
silly.py:
def thing():
print("Wow!")
You can run horrible.py and while it is running you can edit silly.py on disk to make it do something else. When silly.py is finally imported, the updated version will be loaded.
A workaround is to put all your imports at the top of the file, which you probably do anyway.
When a python program is run it is compiled (kinda, more like translated) into a .pyc file that is then run by the python interpreter. When you change a file it should NOT affect the code if it is already running.
Here is a related stackoverflow answer. What will happen if I modify a Python script while it's running?
Why not have another working directory where you make your modifications? Is there a lot of ancillary data or something that makes it hard to set up a working directory? I.e. if your working directory is A, git clone A B, and then work in B. When you're done, you can pull the changes back from B to A:
git remote add B ../B
git pull B master

Run same python code in two terminals, will them interfere each other?

I have a python script which takes a while to finish its executing depending on the passed argument. So if I run them from two terminals with different arguments, do they get their own version of the code? I can't see two .pyc files being generated.
Terminal 1 runs: python prog.py 1000 > out_1000.out
Before the script running on terminal 1 terminate, i start running an another; thus terminal 2 runs: python prog.py 100 > out_100.out
Or basically my question is could they interfere with each other?
If you are writing the output to the same file in disk, then yes, it will be overwritten. However, it seems that you're actually printing to the stdout and then redirect it to a file. So that is not the case here.
Now answer to your question is simple: there is no interaction between two different executions of the same code. When you execute a program or a script OS will load the code to the memory and execute it and subsequent changes to code has nothing to do with the code that is already running. Technically a program that is running is called a process. Also when you run a code on two different terminals there will be two different processes on the OS one for each of them and there is no way for two process to interfere unless you explicitly do that (IPC or inter-process communication) which you are doing here.
So in summary you can run your code simultaneously on different terminals they will be completely independent.
Each Python interpreter process is independent. How the script reacts to itself being run multiple times depends on the exact code in use, but in general they should not interfere.
.pyc file reference http://effbot.org/pyfaq/how-do-i-create-a-pyc-file.htm
Python automatically compiles your script to compiled code, so called
byte code, before running it. When a module is imported for the first
time, or when the source is more recent than the current compiled
file, a .pyc file containing the compiled code will usually be created
in the same directory as the .py file.
If you afraid your code get overwritten due to whatever mistake, you should learn to put your code under VERSION CONTROL. Register github and use git to do that.
bigger sign ">" will send the output to the right handler. It you specify file name, it will push the output to that file name. Even in different terminal, if you run the code inside the same folder, use the ">" point to SAME file name, the file on the right of the ">" definitely get overwrite.
Program SOURCE CODE ARE NOT mutable during execution. Unless you acquire high level program hacking skill.
Each program will run inside its "execution workspace". Unless you make a code that tap into same resources(like change same file,shared reources ), otherwise there is no interference. (except if one exhaust all CPU, Memory resources, the second one will be interfere, but that is other story)

Running a Python Program Hiding the Source

When I run a program from a USB memory, and remove the USB memory the program still goes on running (I mean with out really copying the program into the Windows PC).
However, does the program make its copy inside the Windows in any hidden location or temporary folder while running by the python IDLE. From where the python IDLE receive the code to be running after removing the USB memory? I am going to run python program in a public shared PC so I do not want anyone find out my code, I just want to run it, and get the result next day. Does someone can get my code even I remove the USB memory?
There are plenty of ways someone can get your program, even if you remove the USB drive.
They can install a program that triggers when a USB stick is inserted, search the stick for .py files, and copies them to disk.
If the Python installation you're using is on the disk instead of the USB drive, they can replace the Python executable with a wrapper that saves copies of any file the Python interpreter opens.
Your program is going to go into RAM, and depending on what it does and what else is using the machine, it may get swapped to disk. An attacker may be able to read your program out of RAM or reconstruct it from the swap file.
It sounds like you are doing something you probably shouldn't be doing. Depending on how much people want your code they could go as far as physically freezing the ram and doing a forensic IT analysis. In short, you can't prevent code cloning on a machine you don't administer.

Having problems profiling memory in Python program using Valgrind

I've looked at some questions about profiling memory usage in Python programs, but so far haven't been able to get anything to work. My program must run as root (it opens a TUN/TAP device).
First, I tried heapy; unfortunately this didn't work for me. Every time my code tried to execute hpy().heap() the program froze. Not wanting to waste too much timed I decided to try valgrind.
I tried valgrind with massif:
# valgrind --tool=massif ./my_prog.py --some-options value
I think the issue is related to profiling Python programs. I tried my program (which runs as root) and no massif output file was generated. I also wasn't able to generate an output file with another Python program (which doesn't run as root). However, a simple C test program worked fine and produced the massif file.
What are the issues preventing Valgrind and massif from working correctly with Python programs?
Instead of having the script launch the interpreter, directly calling it as a parameter to Valgrind solves the problem.
valgrind --tool=massif python my_script.py

Categories

Resources