I created a simple python project that serves up a couple of pages. I'm using the 'webapp' framework and django. What I'm trying to do is use one template file, and load 'content files' that contain the actual page text.
When I try to read the content files using os.open, I get the following error:
pageContent = os.open(pageUrl, 'r').read()
OSError: [Errno 1] Operation not permitted: 'content_includes/home.inc' error
If I let the django templating system to read the same file for me, everything works fine!
So the question is What am I doing wrong that django isn't??? The same 'pageUrl' is used.
The code below will give me the error, while if I comment out the first pageContent assignment, everything works fine.
Code:
pageName = "home";
pageUrl = os.path.join(os.path.normpath('content_includes'), pageName + '.inc')
pageContent = os.open(pageUrl, 'r').read()
pageContent=template.render(pageUrl, template_values, debug=True);
template_values = { 'page': pageContent,
'test': "testing my app"
}
Error:
Traceback (most recent call last):
File "/opt/apis/google_appengine/google/appengine/ext/webapp/__init__.py", line 511, in __call__
handler.get(*groups)
File "/home/odessit/Development/Python/Alpha/main.py", line 19, in get
pageContent = os.open(pageUrl, 'r').read()
File "/opt/apis/google_appengine/google/appengine/tools/dev_appserver.py", line 805, in FakeOpen
raise OSError(errno.EPERM, "Operation not permitted", filename)
OSError: [Errno 1] Operation not permitted: 'content_includes/home.inc'
app.yaml:
handlers:
- url: /javascript
static_dir: javascript
- url: /images
static_dir: images
- url: /portfolio
static_dir: portfolio
- url: /.*
script: main.py
os.path.normpath() on "content_includes" is a no-op - normpath just removes double slashes and other denormalizations. What you probably want is to build a path relative to the script, in which case you should do something like os.path.join(os.path.dirname(__file__), 'content_includes', pageName + '.inc').
If you dig in the dev_appserver.py source code and related files you see that the server does some incarnate checking to ensure that you open only files from below your applications root (in fact the rules seem even more complex).
For file access troubled I instrumented that "path permission checking" code from the development server to find that I was using absolute paths. We propbably should do a patch to appengine to provide better error reporting on that: IIRC the Appserver does not display the offending path but a mangled version of this which makes debugging difficult.
Related
I am trying to get remoteconfig working, following this guide:
https://pypi.org/project/remoteconfig/
As a control, I have this code that works:
config.read('./config.ini')
for section in config:
print(section)
When I put the same config file in a remote Gitlab, this code does not work:
from remoteconfig import config
config.read('https://myorg.org/path/repo/~/blob/app/config.ini')
for section in config:
print(section)
What could I be doing wrong here? The error msg I am getting is:
configParser.MissingSectionHeaderError: File contains no section headers
So it seems like it's reaching the file path (network/connectivity OK), but not liking what's in that file or possibly the file format? The same exact file works with localconfig.
For now I am going to use the 'gitlab' pip module and simply consume the API for the file (with private_token:
f = project.files.get(file_path='path/file', ref='master'
I'm trying to run a test in a docker container, which runs locally with no issues:
I want to upload a correct.csv file from 'correct' directory
*** Keyword ***
Upload file
[Arguments] ${directory} ${file}
Choose File ${choose_file_input} ${EXECDIR}/Files/${directory}/${file}
** Test case ***
Upload
Upload file correct correct.csv
But when running test in docker I get a FAIL with the AttributeError: module 'base64' has no attribute 'encodestring'. Is it because there is no GUI in docker? or the encoding needs to be fixed? Or eventually maybe there is another solution I can use for uploading files?
15:20:01.250 INFO Sending /App/Files/correct/correct.csv to browser.
15:20:01.251 DEBUG POST http://192.168.1.29:4444/wd/hub/session/4b6d453b394adaaa51bb4149e9ba8678/elements {"using": "xpath", "value": "//div[#id=\"upload\"]//input"}
15:20:01.252 DEBUG Starting new HTTP connection (1): 192.168.1.29:4444
15:20:01.305 DEBUG http://192.168.1.29:4444 "POST /wd/hub/session/4b6d453b394adaaa51bb4149e9ba8678/elements HTTP/1.1" 200 90
15:20:01.305 DEBUG Finished Request
15:20:01.618 FAIL AttributeError: module 'base64' has no attribute 'encodestring'
15:20:01.619 DEBUG Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/SeleniumLibrary/__init__.py", line 490, in run_keyword
return DynamicCore.run_keyword(self, name, args, kwargs)
File "/usr/local/lib/python3.9/site-packages/robotlibcore.py", line 103, in run_keyword
return self.keywords[name](*args, **(kwargs or {}))
File "/usr/local/lib/python3.9/site-packages/SeleniumLibrary/keywords/formelement.py", line 224, in choose_file
self.find_element(locator).send_keys(file_path)
File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 475, in send_keys
value = self._upload(local_file)
File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/remote/webelement.py", line 695, in _upload
content = base64.encodestring(fp.getvalue())
Based on the traceback you have found this issue:
Selenium 3 is incompatible with Python 3.9
This is the fix for the issue: DeprecationWarning of base64.encodestring().
They won't back port this fix:
Thanks for the issue. We won't be releasing another version 3 as we're
heading to finishing off Selenium 4. It is a drop in replacement to
use Selenium 4.0.0.a5 so should work the same. There should not be any
breaking changes.
So you could upgrade selenium to Selenium 4.0.0.a5 or
Downgrade Python to 3.7 for example. I suppose locally you do not run 3.9.
We were running into this issue as well, but going back to an older version of Python was not an option due to incompatibilities with other libraries. If you find yourself in the same spot, you can re-create the alias like so:
import base64
base64.encodestring = base64.encodebytes
In whatever your entry-point is.
I am reading 'Head First Python'. I am on Chapter 10 where Google App Engine is introduced. The initial hello world of using Python and Google App Engine was successful but subsequent programs have all failed.
I have the following app.yaml file:
application: three
version: 1
runtime: python27
api_version: 1
threadsafe: false
handlers:
- url: /.*
script: page368b.py
libraries:
- name: django
version: "1.3"
With the following Python code (page368b.py):
import wsgiref.handlers
from google.appengine.ext import webapp
from google.appengine.ext import db
from google.appengine.ext.webapp import template
#this line throws the error when accessing the web-page
from google.appengine.ext.db import djangoforms
import birthDB
class BirthDetailsForm(djangoforms.ModelForm):
class Meta:
model = birthDB.BirthDetails
class SimpleInput(webapp.RequestHandler):
def get(self):
html = template.render('templates/header.html', {'title': 'Provide your birth details'})
html = html + template.render('templates/form_start.html', {})
html = html + str(BirthDetailsForm(auto_id=False))
html = html + template.render('templates/form_end.html', {'sub_title': 'Submit Details'})
html = html + template.render('templates/footer.html', {'links': ''})
self.response.out.write(html)
def main():
app = webapp.WSGIApplication([('/.*', SimpleInput)], debug=True)
wsgiref.handlers.CGIHandler().run(app)
if __name__ == '__main__':
main()
Here is another Python module imported into the one above called birthDB.py:
from google.appengine.ext import db
class BirthDetails(db.Model):
name = db.StringProperty()
date_of_birth = db.DateProperty()
time_of_birth = db.TimeProperty()
There is a templates folder that the above Python module calls. They have HTML code in them with some Django code. Here is an example using the footer.html.
<p>
{{ links }}
</p>
</body>
</html>
The other HTML files are similar. I can start the Google App Engine with no problems using this command from BASH: python google_appengine/dev_appserver.py ~/Desktop/three The directory three contains the templates folder, the app.yaml file, the Python modules shown above.
My problem is when I access the web-page at http://localhost:8080 nothing is there and the BASH shell where the command is run to start this shows all the calls in the Python program that caused the problem and then finally says: ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
I have read in a few different places a few different things to try so I thought I would go ahead and make a new post and hope that some expert Python programmers would chime in and lend their assistance to a lost hobbit such as myself.
Also, the book says to install Python2.5 to use this code but Google App Engine now supports Python2.7 that was not available as of the time of the books writing. Also, I just checked and Python2.5 is not even an option to use with Google App Engine. Python2.5 deprecated
This is probably too complex to solve on here. I am surprised all of these different technologies are used in a Head First Python book. It is asking a lot of a Python noob. ^_^
Regards,
user_loser
UPDATE - I installed Django on my Ubuntu Operating System
When I change the line in the python module 368b.py from google.appengine.ext.db import djangoforms to from django import forms I receive the following error on the console when accessing the web-page on localhost:
loser#loser-basic-box:~/Desktop$ google_appengine/dev_appserver.py three
INFO 2014-09-06 21:08:36,669 api_server.py:171] Starting API server at: http://localhost:56044
INFO 2014-09-06 21:08:36,677 dispatcher.py:183] Starting module "default" running at: http://localhost:8080
INFO 2014-09-06 21:08:36,678 admin_server.py:117] Starting admin server at: http://localhost:8000
ERROR 2014-09-06 21:08:48,942 cgi.py:121] Traceback (most recent call last):
File "/home/loser/Desktop/three/page368b.py", line 13, in <module>
class BirthDetailsForm(forms.ModelForm):
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/forms/models.py", line 205, in __new__
opts.exclude, opts.widgets, formfield_callback)
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/forms/models.py", line 145, in fields_for_model
opts = model._meta
AttributeError: type object 'BirthDetails' has no attribute '_meta'
INFO 2014-09-06 21:08:48,953 module.py:652] default: "GET / HTTP/1.1" 500 -
ERROR 2014-09-06 21:08:49,031 cgi.py:121] Traceback (most recent call last):
File "/home/loser/Desktop/three/page368b.py", line 13, in <module>
class BirthDetailsForm(forms.ModelForm):
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/forms/models.py", line 205, in __new__
opts.exclude, opts.widgets, formfield_callback)
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/forms/models.py", line 145, in fields_for_model
opts = model._meta
AttributeError: type object 'BirthDetails' has no attribute '_meta'
Update Errors from running the program as is without making any changes:
loser#loser-basic-box:~/Desktop$ google_appengine/dev_appserver.py three/
INFO 2014-09-06 21:35:19,347 api_server.py:171] Starting API server at: http://localhost:60503
INFO 2014-09-06 21:35:19,356 dispatcher.py:183] Starting module "default" running at: http://localhost:8080
INFO 2014-09-06 21:35:19,358 admin_server.py:117] Starting admin server at: http://localhost:8000
ERROR 2014-09-06 21:35:25,011 cgi.py:121] Traceback (most recent call last):
File "/home/loser/Desktop/three/page368b.py", line 13, in <module>
class BirthDetailsForm(djangoforms.ModelForm):
File "/home/loser/Desktop/google_appengine/google/appengine/ext/db/djangoforms.py", line 772, in __new__
form_field = prop.get_form_field()
File "/home/loser/Desktop/google_appengine/google/appengine/ext/db/djangoforms.py", line 370, in get_form_field
return super(DateProperty, self).get_form_field(**defaults)
File "/home/loser/Desktop/google_appengine/google/appengine/ext/db/djangoforms.py", line 353, in get_form_field
return super(DateTimeProperty, self).get_form_field(**defaults)
File "/home/loser/Desktop/google_appengine/google/appengine/ext/db/djangoforms.py", line 200, in get_form_field
return form_class(**defaults)
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/forms/fields.py", line 340, in __init__
super(DateField, self).__init__(*args, **kwargs)
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/forms/fields.py", line 99, in __init__
widget = widget()
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/forms/widgets.py", line 382, in __init__
self.format = formats.get_format('DATE_INPUT_FORMATS')[0]
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/utils/formats.py", line 67, in get_format
if use_l10n or (use_l10n is None and settings.USE_L10N):
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/utils/functional.py", line 276, in __getattr__
self._setup()
File "/home/loser/Desktop/google_appengine/lib/django-1.3/django/conf/__init__.py", line 40, in _setup
raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
I assume you're working with the example code that accompanies the book, available from this website: http://examples.oreilly.com/0636920003434/
If you download and expand the chapter 10 archive (chapter10.zip), you'll see several example files, and also several .zip archives. The page368b.py file corresponds with the webapp-chapter10-simpleform.zip archive. Open that archive to create a webapp-chapter10-simpleform directory. This directory contains an app.yaml file, the simpleform.py file (identical to page368b.py), birthDB.py (an ext.db model class), and static and template file directories.
Unfortunately, as you may have already noticed, the example doesn't work out of the box with the latest SDK. (I'm using version 1.9.10 of the SDK, which was just released.) It reports "ImportError: No module named django.core.exceptions" when you attempt to load the page. Strictly speaking, this example is not a Django application, but is merely trying to use a library that depends on Django being present.
The Python 2.5 runtime environment, which is selected by the app.yaml file included with this example, is meant to include Django 0.96 by default. However, this behavior has changed in the SDK since Head First Python was written. The smallest fix to get this example to work is to add these lines to simpleform.py prior to the import of djangoforms:
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from google.appengine.dist import use_library
use_library('django', '0.96')
Then create a file named settings.py in your application root directory (the webapp-chapter10-simpleform directory). This file can be empty in this case. (As other commenters have noted, there is a better way to generate this file when using the Django framework, but in this case we just need the import to succeed.)
To upgrade this example to use the Python 2.7 runtime environment, modify app.yaml as follows:
application: simpleform
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /static
static_dir: static
- url: /*.
script: simpleform.app
The changes:
runtime: is now python27
threadsafe: true is added
The reference to simpleform.py is now a Python object path to the global variable app.
Then modify simpleform.py so that everything following the definition of the SimpleInput class is replaced with:
app = webapp.WSGIApplication([('/.*', SimpleInput)], debug=True)
Instead of running the simpleform.py script, the Python 2.7 runtime environment imports it as a module, then looks for a WSGI application object in a global variable. The lines we removed executed the app, and that's now done for us by the runtime environment.
From here, you can use a libraries: clause as you have done to select a newer version of Django, if you wish. A quick test in a dev server shows that the modified example works with Django 1.5, which is the latest supported version.
I'm trying to download an MP3 file, via its URL, using Python's urllib2.
mp3file = urllib2.urlopen(url)
output = open(dst,'wb')
output.write(mp3file.read())
output.close()
I'm getting a urllib2.HTTPError: HTTP Error 403: Forbidden error.
Trying urllib also fails, but silently.
urllib.urlretrieve(url, dst)
However, if I use wget, I can download the file successfully.
I've noted the general differences between the two methods mentioned in "Difference between Python urllib.urlretrieve() and wget", but they don't seem to apply here.
Is wget doing something to negotiate permissions that urllib2 doesn't do? If so, what, and how do I replicate this in urllib2?
Could be something on the server side - blocking python user agent for example. Try using wget user agent : Wget/1.13.4 (linux-gnu) .
In Python 2:
import urllib
# Change header for User-Agent
class AppURLopener(urllib.FancyURLopener):
version = "Wget/1.13.4 (linux-gnu)"
url = "http://www.example.com/test_file"
fname = "test_file"
urllib._urlopener = AppURLopener()
urllib.urlretrieve(url, fname)
The above didn't work for me (I'm using python3.5). wget works fine.
It's not (I assume) a huge problem for me - surely I can still do a system() and use wget to get the data, with some file renaming and munging.
But in case anyone else is suffering from the same problem, these are the errors I get from the above snippet:
Traceback (most recent call last):
File "./mksynt.py", line 10, in <module>
class AppURLopener(urllib.FancyURLopener):
AttributeError: module 'urllib' has no attribute 'FancyURLopener'
I see that the original answer was only promised to work in python2.
I've been trying to understand how the application_readable static url handler field works. I'm using SDK version 1.7.7 and I've set this to true on an application in my dev environment, but I can't seem to actually read the file:
# app.yaml
- url: /test
static_dir: application/static/test
application_readable: true
# app.py
path = os.path.join(os.path.split(__file__)[0], 'static/test/test.png')
self.response.out.write('Looking for %s...' % path)
self.response.out.write(os.path.exists(path))
self.response.out.write("\n")
path = '/application/static/test/test.png'
self.response.out.write('Looking for %s...' % path)
self.response.out.write(os.path.exists(path))
self.response.out.write("\n")
path = 'application/static/test/test.png'
self.response.out.write('Looking for %s...' % path)
self.response.out.write(os.path.exists(path))
self.response.out.write("\n")
path = '/static/test/test.png'
self.response.out.write('Looking for %s...' % path)
self.response.out.write(os.path.exists(path))
self.response.out.write("\n")
path = 'static/test/test.png'
self.response.out.write('Looking for %s...' % path)
self.response.out.write(os.path.exists(path))
self.response.out.write("\n")
path = '/test/test.png'
self.response.out.write('Looking for %s...' % path)
self.response.out.write(os.path.exists(path))
self.response.out.write("\n")
path = 'test/test.png'
self.response.out.write('Looking for %s...' % path)
self.response.out.write(os.path.exists(path))
self.response.out.write("\n")
path = '/test.png'
self.response.out.write('Looking for %s...' % path)
self.response.out.write(os.path.exists(path))
self.response.out.write("\n")
path = 'test.png'
self.response.out.write('Looking for %s...' % path)
self.response.out.write(os.path.exists(path))
But none of those work:
Looking for /vagrant/test/application/static/test/test.png...False
Looking for /application/static/test/test.png...False
Looking for application/static/test/test.png...False
Looking for /static/test/test.png...False
Looking for static/test/test.png...False
Looking for /test/test.png...False
Looking for test/test.png...False
Looking for /test.png...False
Looking for test.png...False
Though the file definitely exists:
vagrant#precise64:/vagrant/kissyface$ ls -l /vagrant/test/application/static/test/test.png
-rwxrwxrwx 1 vagrant vagrant 9920 May 3 18:13 /vagrant/test/application/static/test/test.png
Can anyone tell me what I'm doing wrong? I haven't been able to find any documentation or example code for this beyond the brief description in the statis url handler docs and a mention in the appengine sdk 1.7.6 changelog. Is there a utility class that provides access to these files, or am I completely misinterpreting what application_readable is actually supposed to do?
Overview
It's somewhat difficult to reason about what exactly is going on on your system, but I can tell you what works on mine. We can speculate all day about what could be wrong, but implementing something that works and working backwards is usually more productive than guessing; it could be anything.
If I were to guess, I'd say that the problem is:
Incorrect handler order
Screwed up python paths
Screwed up python versions
Using a janky old SDK
Underpants gnomes
If, instead of guessing, you implement the project I've outlined below, it should be pretty simple to reason backward and find out why it's not working for you. If this project doesn't work, you've got some work to do. The problem's really nasty (and I don't want to help you fix it). If it does work, you're in luck, since you're 5 or 10 minutes away from having the rest of your code working too!
Using the latest python appengine SDK from http://code.google.com/p/googleappengine/downloads/list:
google_appengine_1.8.0.zip
71b5f3ee06dce0a7d6af32d65ae27272eac038cb
Project files and their contents:
Directory Setup:
.
├── app.py
├── app.pyc
├── app.yaml
└── static
└── hi.txt
app.py:
import webapp2
import os
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('Hello, webapp World!\n\n')
path = os.path.join(os.path.split(__file__)[0], 'static/hi.txt')
self.response.out.write(open(path).readlines()[0])
application = webapp2.WSGIApplication([('/.*', MainPage)])
app.pyc is the (automatically) compiled version of this file.
app.yaml:
application: myapp
version: 1
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: /static
static_dir: static
application_readable: true
- url: /.*
script: app.application
static/hi.txt:
Ezra can see this text fine; I'm not sure why you can't... Hi!
What happens:
Start the webserver from the project root:
dev_appserver.py --port 80 .
You might have to use a different port number; it's no big deal. Just adjust the instructions that follow for the one you choose.
Visiting http://localhost/ in the browser:
Http response:
INFO 2013-05-14 09:45:57,372 server.py:585] default: "GET / HTTP/1.1" 200 85
Browser output:
Hello, webapp World!
Ezra can see this text fine; I'm not sure why you can't... Hi!
Visiting http://localhost/static/hi.txt in the browser:
Http response:
INFO 2013-05-14 09:48:42,785 server.py:585] default: "GET /static/hi.txt HTTP/1.1" 200 63
Browser output:
Ezra can see this text fine; I'm not sure why you can't... Hi!
Breaking it:
If I remove the application_readable: true line from app.yaml:
Visiting http://localhost/ in the browser:
Http Response:
ERROR 2013-05-14 09:51:13,290 webapp2.py:1528] [Errno 13] file not accessible: '.../static/hi.txt'
Browser output:
500 Internal Server Error
The server has either erred or is incapable of performing the requested operation.
What to do next:
Hopefully you can work backwards from this example. If it doesn't work for you, you're doomed. Enjoy spending a sunny mid-May afternoon trawling through paths and trying to (re/un)install things to get this frickin' thing going. The list at the top is a list of I'd try, without knowing any specifics and having ruled out programming error. Good luck!