Develop Reference

How to import a local python module when using the sbatch command in SLURM

I was using the cluster manager slurm and I was running a submission script with sbatch (with a python interpeter). The sbatch submission imported one of my modules called main_nn.py. The module is located in the same place as my submission directory, however, python fails to find it even though the file exists. I am having a hard time figuring it out why this is happening. My python file looks as follow:
#!/usr/bin/env python
#SBATCH --job-name=Python
print('hi')
import main_nn
however the output of my slurm dump file is:
hi
Traceback (most recent call last):
File "/home/slurm/slurmd/job3223398/slurm_script", line6, in <module>
import main_nn
ImportError: No module named main_nn
I tried checking if the module main_nn was in the current directory and it was there indeed. Thus, the first thing that seemed suspicious to me was that the error in the slurm file said the location of my script was at "/home/slurm/slurmd/job3223398/slurm_script" rather than at path_to_project. Thus I went ahead an added the line
os.system('pwd')
to see where my script was executing from and to my surprise it was executing at path_to_project rather than at "/home/slurm/slurmd/job3223398/slurm_script" which must mean that sbatch is doing something funky to executed a script at one location but make it think its at another. If this is the case how am I suppose to do an import in python where the module is in the same location as in my submission script? Am I forced to put it in a package and trick python to think its in a package/library?

As Slurm copies the submission script to a specific location on the compute node to run it, your Python script will not find the modules that are in the submission directory.
But Slurm correctly sets the current working directory so you can explicitly add it to the python path with something like:
sys.path.append(os.getcwd())
near the beginning of your script.

cx_Oracle, and Library paths

Pretty new to all this so I apologize if I butcher my explanation. I am using python scripts on a server at work to pull data from our Oracle database. Problem is whenever I execute the script I get this error:
Traceback (most recent call last):
File "update_52w_forecast_from_oracle.py", line 3, in
import cx_Oracle
ImportError: libnnz11.so: cannot open shared object file: No such file or direct ory
But if I use:
export LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib
Before executing the script it runs fine but only for that session. If I log back in again I have to re-set the path. Anything I can do to make this permanent? I'm trying to use Cron as well to automate the script once a week. It was suppose to automate early Monday morning but it didn't run.
EDIT: Just had to add the path to my .bashrc file in the root directory.

Well, that was pretty simple. I just had to add it to the .bashrc file in my root directory.

How to use websockify with python on windows or compiling websockify as windows executable (for more multiple client connection at a time)

I'm trying to get the websockify 0.6.0 running on windows but without any luck,
I have tried python websockify.py 1501 10.0.0.141:1501 but its not good, getting errors , like that:
Traceback (most recent call last):
File "websockify.py", line 1, in <module>
run
NameError: name 'run' is not defined
I also tried Compiling Websockify as Windows Executable , but this also didn't work
I use the following command run.exe 1501 10.0.0.141:1501 and it looks promising at the beginning, outputting the following to the console:
WARNING: no 'resource' module, daemonizing is disabled
WebSocket server settings:
- Listen on :1501
- Flash security policy server
- No SSL/TLS support (no cert file)
- proxying from :1501 to 10.0.0.141:1501
but then after trying to connect using the following from the browser ws://localhost:1501
**it outputs the following error
Traceback (most recent call last):
File "run", line 5, in <module>
File "websockify\websocketproxy.pyc", line 419, in websockify_init
File "websockify\websocket.pyc", line 1018, in start_server
UnboundLocalError: local variable 'exc' referenced before assignment
Any idea on how to use the websockify on windows / or how to use the compiled websockify as windows executable ?

The easiest way to get websockify working on Windows is to use the Node.js version of websockify (in the other/js directory). It works perfectly out of the box, with no shenanigans required.

To address this, use the modified following commands for your example source, start from the beginning of each step and see if it helps:
Firstly, install Portable Python 2.7
You then need to modify the setup.py (It looks like this is why you are getting your first error, as you may not have defined run):
from setuptools import setup, find_packages
# add (line 2):
import py2exe
setup(name=name,
# add (line 10):
console=['run'],
Ensure the above has executed correctly by inspecting setup.py and ensure it includes run.
In your local code, import the resources module to allow you to monitor, measure and control system resources utilized by your program
import resource
Inspect your local variable exc and ensure you have assigned a value to it before calling it (I'm guessing you may have attributed a system variable to it, but python was unable to do so as you did not have resources imported, and as such it was not assigned). If you like, put up an example of your code in the comment to this response and I'll take a closer look at this part.
Back to your source guide, navigate to the websockify folder in command prompt, then execute the following to compile websockify:
[Your path to Portable Python install]\App\python.exe setup.py py2exe --includes numpy
You will now see in the websockify directory a new dir 'dist' which contains the compiled exe. An example provided is:
run.exe 5901 localhost:5900
There is also a guide here to run websockify as a Windows Service if this suits (again mentioned in your source).
----Further edit for more detail----
Open up the two files that seem to be giving you issues (websockify\websocketproxy.pyc and websockify\websocket.pyc and ensure that any reference to a variable called "exc" is referenced after it has been assigned a value (small chance of an issue if you have not yet modified these files.
I believe that your code is relying upon making and monitoring changes to the system resources (such as ports etc) and you are not allowing your code to have these permissions, so it needs the resources module. If you are calling run.exe from a program (what I called your local code) then you need to import resources at the top. If you are just straight up calling the run.exe program from a command line, then try making this new program and see if this helps. If not, send me the contents of your websockify folder and run.exe and I will take a look
# Program Name: goAndRun.py
# Code:
import sys, string, os, arcgisscripting, resource
os.chdir( 'C:\\[Path_to_the_dir_containing_run.exe]' )
os.system( '"C:\\[Path_to_the_dir_containing_run.exe]\run.exe, 5901 localhost:5900"' )
And then use the command:
python goAndRun.py
Not being in your environment, I cannot tell if this will execute exactly as I have written it. The last line may also be:
os.system( '"C:\\[Path_to_the_dir_containing_run.exe]\run.exe"', '5901 localhost:5900' )

Python scripts hang on exception until ctrl-c

I have an issue with running python scripts on one server running RH Linux with Python 2.6.6. When I run any script and it produces an exception, the script hangs until I press CTRL-C and then it prints the traceback information. It happens when I run the script from the command-line without calling python directly, making use of the shebang on the first line of the script. If I execute the script by calling python , I don't have the same hanging issue. I've searched and seen similar issues, but they were related to specific libraries where this happens with all python scripts. I've tried a different servers with the same script and didn't have the issue. I have included a simple script that I have been testing with. It tries to open a non-existent file.
#!/usr/bin/env python
tempfile = open('noexists.txt','r')
When I execute the code as "test.py" on the command-line, I get the following response:
~/bin$> test.py
^CTraceback (most recent call last):
File "/export/home/jwd3/bin/test.py", line 2, in <module>
tempfile = open('noexists.txt','r')
IOError: [Errno 2] No such file or directory: 'noexists.txt'
If I execute it as "python test.py" then I get the following response:
~/bin$> python test.py
Traceback (most recent call last):
File "test.py", line 2, in <module>
tempfile = open('noexists.txt','r')
IOError: [Errno 2] No such file or directory: 'noexists.txt'
The difference is difficult to tell here, but it is very evident when executing. Look at the first sample output and notice the "^C" before the Traceback. Until I hit CTRL-C, the script was just hanging. The second sample output returns the traceback of the exception imediately without hanging.
I've tried moving the script to a new location, changing the shebang from #!/usr/bin/python to #!/usr/bin/env python and different scripts. In all cases they behave the same. Any help would be greatly appreciated. I do not want to call all python scripts using "python " format.

Thanks to the help from the suggestions above, I was able to find the solution to the issue. After running strace, I found the code was hanging on a socket connection. Searching for the same socket connection info, I found another issue already posted on Stack Overflow that was similar. I then found a second issue that was also very similar. Both issues had the same suggestion. Apparently the Python script was conflicting with a service called abrtd. The suggestion was to restart this service (abrtd). After restarting it, the issue with the Python script hanging was resolved.

PYTHONPATH error Linux Terminal v Console

I am having trouble getting PYTHONPATH set. I have added this line to ~/.bashrc
export PYTHONPATH=$PYTHONPATH:/home/sayth/My_Programs/Python
I ran source ~/.bashrc
Here is the issue though if I type python at the terminal and get a python REPL I can then import modules from ~/My_Programs/Python without a problem.
However if I run bpython or idle or a console in ninja-ide etc then import fails.
>>> import temperature
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
import temperature
ImportError: No module named temperature
>>>
I can do sys.path.append(/home/sayth/My_Programs/Python) and get it working, but I wanted the PYTHONPATH to work in all consoles automatically. How?
PS using ubuntu 11.10

I'm going to guess that you did not launch your ninja-ide from the same terminal that you source ~/.bashrc from -- the new environment variable only influences child processes of whichever shell process did the sourcing. Of course, logging out and in again may do it, but if you're living entirely in the GUI (e.g., you click a button to start the ninja-ide), then you may need to go to some more effort to set your environment variable. How you do so depends upon which graphical environment you're using.