Develop Reference

Get the output of .exe file in Python3 on Windows

I've been trying to get the output of the following command 'manage-bde -status' which works fine on my windows cmd commande prompt, but by using a python program.The same command doesn't work on python subprocess, so I had to launch the manage-bde.exe file.
So my code looks like this now :
import os, platform, subprocess
################This part is only helpful for resolving 32 bit/64 bits issues##########
system_root = os.environ.get('SystemRoot', 'C:\\Windows');
if (platform.architecture()[0] == '32bit' and platform.machine() == 'AMD64'):
system32='Sysnative'
else:
system32='System32'
manage_bde = os.path.join(system_root, system32, 'manage-bde.exe')
print(manage_bde)
#######################################################################################
stdout=subprocess.check_output(['start',manage_bde,'-status'])
print('Output:'+stdout)
I'm launching it from cmd commande line, with Python 3.7.0. The problem is that I get the following output :
C:\WINDOWS\Sysnative\manage-bde.exe (this is from my print(manage_bde))
Traceback (most recent call last):
File "testCLI.py", line 13, in <module>
stdout=subprocess.check_output(['start',manage_bde,'-status'])
File "d:\Profiles\user\AppData\Local\Programs\Python\Python3\lib\subprocess.py", line 376, in check_output
**kwargs).stdout
File "d:\Profiles\user\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 453, in run
with Popen(*popenargs, **kwargs) as process:
File "d:\Profiles\user\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 756, in __init__
restore_signals, start_new_session)
File "d:\Profiles\user\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 1155, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] Specified file was not found
I launch it from the D: Drive. Does anyone know what I missed ?

You probably need to run the script with administrator privileges.
If the file exists in the filesystem and you get FileNotFoundError: [WinError 2], you can test whether the the script has enough permissions to see the file with:
os.stat(manage_bde)

Yes I have enough permissions to see the file : the os.stat(manage_bde) is working.
I have changed my parameters like eryksun proposed :
stdout = subprocess.getoutput([manage_bde, '-status'])
Now I can launch the program (but only with a shell with administrator rights) and get the output, thank you !

bazel-bin hosted python process seemingly can't find its hosting zip file

I am trying to get a TensorFlow example (textsum) to run, and the readme.md specifies to use bazel to do the training. Ok, I dutifully installed chocolaty and then bazel, and then bazel ran and created the build files successfully. So bazel seems to work.
However when I got to the train step - also orchestrated by bazel - it choked with an IOError message that I don't really understand and can't find much information about.
Here it is:
UXIE+mike#uxie /d/tensorflow/models
$ bazel-bin/textsum/seq2seq_attention --mode=train --article_key=article --abstract_key=abstract
--data_path=data/training-* --vocab_path=data/vocab --log_root=textsum/log_root --train_dir=textsum/log_root/train
Traceback (most recent call last):
File "C:\Python27\lib\runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\Python27\lib\runpy.py", line 72, in _run_code
exec code in run_globals
File "bazel-bin\textsum\seq2seq_attention\__main__.py", line 168, in <module>
File "bazel-bin\textsum\seq2seq_attention\__main__.py", line 115, in Main
File "bazel-bin\textsum\seq2seq_attention\__main__.py", line 98, in CreateModuleSpace
File "C:\Python27\lib\zipfile.py", line 756, in __init__
self.fp = open(file, modeDict[mode])
IOError: [Errno 2] No such file or directory: '\\\\?\\bazel-bin\\textsum\\seq2seq_attention'
I am running this on Windows 10 in a bash shell (have tried a couple - same results everytime) and it is clearly successfully starting the python code contained in the seq2set_attention zip file (see the 2nd line that invokes that file).
However looking at the error dump it seems to get confused and suddenly it can't find that zip file anymore (see the last line).
So a few questions:
I don't remember seeing that \\\\?\\ prefix before. Googling for that string didn't help. I would like to know what it means exactly.
Why can't the executable suddenly not find the zip file it is hosted in? (if that is indeed what is happening)
Finally if you have an idea of what configuration step I might have missed, I would be grateful for any hints.

This is a bug in Bazel 0.4.5, see https://github.com/bazelbuild/bazel/issues/2708
You can build Bazel from HEAD or upgrade it to 0.5.0 release candidate to fix this.
choco install bazel --version 0.5.0-rc6

Failure during project creation using 'djangocms' command

Django CMS 3.4.1
Django 1.10.2
pip modules: http://pastebin.com/vJWvZVfA
pip 8.1.2
Python 2.7.12
OS: Amazon Linux AMI release 2016.03
I'm new to Django CMS, Django, and Python. My previous CMS experience has been with WordPress and I'm try to try out Django CMS as an alternative to WordPress, but am getting an error that I can not seem to figure out. I have updated the pip modules and installed others as suggested in threads on StackOverflow, but to no avail.
I followed the tutorial the instructions, with the exception of the djangocms command, which is incorrect in the tutorial. (It's missing the -w flag.)
EDIT: As requested, here is the output with the --verbose flag added:
(env)[ec2-user#(redacted) tutorial-project]$ djangocms -d postgres://(redacted) -e no --permissions yes -l "en-CA, en-US, en, fr-CA, fr-FR" -p . --starting-page yes --verbose --utc mysite
Creating the project
Please wait while I install dependencies
Package install command: install django-cms<3.5 djangocms-admin-style>=1.2,<1.3 django-treebeard>=4.0,<5.0 psycopg2 djangocms-text-ckeditor>=3.2.1 djangocms-link>=1.8 djangocms-style>=1.7 djangocms-googlemap>=0.5 djangocms-snippet>=1.9 djangocms-video>=2.0 djangocms-column>=1.6 easy_thumbnails django-filer>=1.2 cmsplugin-filer>=1.1 Django<1.9 pytz django-classy-tags>=0.7 html5lib>=0.999999,<0.99999999 Pillow>=3.0 django-sekizai>=0.9 django-select2<5.0six
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-duWZSN/psycopg2/
The installation has failed.
*****************************************************************
Check documentation at https://djangocms-installer.readthedocs.io
*****************************************************************
Traceback (most recent call last):
File "/home/ec2-user/env/bin/djangocms", line 11, in <module>
sys.exit(execute())
File "/home/ec2-user/env/local/lib/python2.7/site-packages/djangocms_installer/main.py", line 33, in execute
verbose=config_data.verbose
File "/home/ec2-user/env/local/lib/python2.7/site-packages/djangocms_installer/install/__init__.py", line 91, in requirements
output = subprocess.check_output(['pip'] + args)
File "/usr/lib64/python2.7/subprocess.py", line 574, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '[u'pip', u'install', u'django-cms<3.5', u'djangocms-admin-style>=1.2,<1.3', u'django-treebeard>=4.0,<5.0', u'psycopg2', u'djangocms-text-ckeditor>=3.2.1', u'djangocms-link>=1.8', u'djangocms-style>=1.7', u'djangocms-googlemap>=0.5', u'djangocms-snippet>=1.9', u'djangocms-video>=2.0', u'djangocms-column>=1.6', u'easy_thumbnails', u'django-filer>=1.2', u'cmsplugin-filer>=1.1', u'Django<1.9', u'pytz', u'django-classy-tags>=0.7', u'html5lib>=0.999999,<0.99999999', u'Pillow>=3.0', u'django-sekizai>=0.9', u'django-select2<5.0six']' returned non-zero exit status 1
EDIT 2: (moved to answer)
EDIT 3:
After solving the first part, another error remains. I'll include it in this post since it still falls under the topic of the title:
Creating the project
Project creation command: /home/ec2-user/env/bin/python2.7 /home/ec2-user/env/bin/django-admin.py startproject mysite /home/ec2-user/tutorial-project
Database setup commands: /home/ec2-user/env/bin/python2.7 -W ignore manage.py migrate
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
utility.execute()
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 346, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
output = self.handle(*args, **options)
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 93, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 19, in __init__
self.loader = MigrationLoader(self.connection)
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 47, in __init__
self.build_graph()
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 191, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 59, in applied_migrations
self.ensure_schema()
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 49, in ensure_schema
if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 162, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 135, in _cursor
self.ensure_connection()
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 118, in connect
conn_params = self.get_connection_params()
File "/home/ec2-user/env/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 158, in get_connection_params
"settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the NAME value.
The installation has failed.
*****************************************************************
Check documentation at https://djangocms-installer.readthedocs.io
*****************************************************************
Traceback (most recent call last):
File "/home/ec2-user/env/bin/djangocms", line 11, in <module>
sys.exit(execute())
File "/home/ec2-user/env/local/lib/python2.7/site-packages/djangocms_installer/main.py", line 41, in execute
django.setup_database(config_data)
File "/home/ec2-user/env/local/lib/python2.7/site-packages/djangocms_installer/django/__init__.py", line 388, in setup_database
output = subprocess.check_output(command, env=env)
File "/usr/lib64/python2.7/subprocess.py", line 574, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/home/ec2-user/env/bin/python2.7', u'-W', u'ignore', u'manage.py', u'migrate']' returned non-zero exit status 1
I'm going to guess that the problem lies in the message:
settings.DATABASES is improperly configured. Please supply the NAME value.
The installation has failed.
I'm using PostgreSQL, so I assume that the error is caused by the schema not being provided in the command argument. There was nothing in the guide that I saw indicating how to provide it, so I'll have to search some more.

The error message tells me absolutely nothing. It only tells me that Django tried to use pip to install something and for whatever reason it failed. In order to figure out what went wrong, try enabling verbose mode (python -v). If that gives the same output, or that isn't an option because you're using a program that calls Python (which it sounds like you are), let me know and prepare to go into the source code. Trust me, I have taken many a dive into source code for Python utilities, and as long as you don't do anything stupid (like delete a file) it's pretty hard to mess anything up. Even if you do you can always reinstall Django (or make a backup first). Don't worry about messing up your computer. You'd have to work hard to make Django do that!

Original error
The non-verbose error message indicates the error in a cryptic way, and using --verbose provides a more understandable message.
I was missing one required module (psycopg2) and four others were versions which were too new (easy_thumbnails, Django, html5lib, django-select2).
I uninstalled the four modules and then reinstalled them using the version requirement in the command. e.g. pip install 'Django<1.10'.
psycopg2 required extra steps, as I got an error message when trying to install it indicating that it could not find pg_config. Another SO answer advised that I just had to install postgresql-devel. After I did that, (sudo yum install postgresql-devel), I also uninstalled postgresql8-libs, as postgresql-devel came with a newer version. I didn't do it before since yum indicated a PHP dependency, and I wasn't sure if it would cause problems if I did.
Second error
Pretty simple. I was missing the /NAME component of the database URL (postgres://USER:PASSWORD#HOST:PORT/NAME).

Python Error while building code using waf

I had code that worked fine. I shut down my PC (Ubuntu) and then, built the same software (ns-3) and now I get the error:
Traceback (most recent call last):
File "./waf", line 148, in <module>
Scripting.prepare(t, cwd, VERSION, wafdir)
File "/home/ns-allinone-3.6/ns-3.6/.waf-1.5.8-12763e767c863088b8579dbeeb8265b6/wafadmin/Scripting.py", line 102, in prepare
prepare_impl(t,cwd,ver,wafdir)
File "/home/ns-allinone-3.6/ns-3.6/.waf-1.5.8-12763e767c863088b8579dbeeb8265b6/wafadmin/Scripting.py", line 95, in prepare_impl
main()
File "/home/ns-allinone-3.6/ns-3.6/.waf-1.5.8-12763e767c863088b8579dbeeb8265b6/wafadmin/Scripting.py", line 130, in main
fun(ctx)
File "/home/ns-allinone-3.6/ns-3.6/.waf-1.5.8-12763e767c863088b8579dbeeb8265b6/wafadmin/Scripting.py", line 269, in build
bld=check_configured(bld)
File "/home/ns-allinone-3.6/ns-3.6/.waf-1.5.8-12763e767c863088b8579dbeeb8265b6/wafadmin/Scripting.py", line 219, in check_configured
bld.load_dirs(proj[SRCDIR],proj[BLDDIR])
File "/home/ns-allinone-3.6/ns-3.6/.waf-1.5.8-12763e767c863088b8579dbeeb8265b6/wafadmin/Build.py", line 245, in load_dirs
self.load()
File "/home/ns-allinone-3.6/ns-3.6/.waf-1.5.8-12763e767c863088b8579dbeeb8265b6/wafadmin/Build.py", line 78, in load
if f:data=cPickle.load(f)
EOFError
I am just amazed at this, how could 2 minutes before everything was fine and now I am screwed by this error.
What should I do, I am totally bewildered. I have a deadline and suddenly this code stops, it worked fine, only change I did was to switch off my PC.

NS-3.6 is a very very old ns-3 release which you are trying to build with a new system (gcc compiler, python etc). I think the only solution to solve the problem is to update the simulator to a new release.
Try to rebuild it. First write: rm -rf build and build from scratch

Change Mapreduce intermediate output location using MRJob

I am trying to run a python script using MRJob on a cluster in which I don't have admin permissions and I got the error pasted below. What I think is happening is that the job is trying to write the intermediate files to the default /tmp.... dir and since this is a protected directory to which I don't have permission to write, the job receives an error and exits. I would like to know how I can change this tmp output directory location to someplace in my local filesystem example:
/home/myusername/some_path_in_my_local_filesystem_on_the_cluster , basically I would like to know what additional parameters I would have to pass to change the intermediate output location from /tmp/... to some place local where I have write permission.
I invoke my script as:
python myscript.py input.txt -r hadoop > output.txt
The error:
no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
creating tmp directory /tmp/13435.1.all.q/mr_word_freq_count.myusername.20131215.004905.274232
writing wrapper script to /tmp/13435.1.all.q/mr_word_freq_count.myusername.20131215.004905.274232/setup-wrapper.sh
STDERR: mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: user=myusername, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
Traceback (most recent call last):
File "/home/myusername/privatemodules/python/examples/mr_word_freq_count.py", line 37, in <module>
MRWordFreqCount.run()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/job.py", line 500, in run
mr_job.execute()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/job.py", line 518, in execute
super(MRJob, self).execute()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/launch.py", line 146, in execute
self.run_job()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/launch.py", line 207, in run_job
runner.run()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/runner.py", line 458, in run
self._run()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/hadoop.py", line 236, in _run
self._upload_local_files_to_hdfs()
File "/home/myusername/.local/lib/python2.7/site-packages/mrjob/hadoop.py", line 263, in _upload_local_files_to_hdfs
self._mkdir_on_hdfs(self._upload_mgr.prefix)

Are you running mrjob as a "local" job, or trying to run it on your Hadoop cluster?
If you are actually trying to use it on Hadoop, you can control the "scratch" HDFS location (where mrjob will store intermediate files) using the --base-tmp-dir flag:
python mr.py -r hadoop -o hdfs:///user/you/output_dir --base-tmp-dir hdfs:///user/you/tmp hdfs:///user/you/data.txt