I've tried to create my own Pypi repository, respecting https://www.python.org/dev/peps/pep-0503/. My idea was to put it in a shared directory (I'm using Windows), say host1/my-pypi. I've generated the index.html needed:
(dir) host1\my-pypi
-> (dir) toto
(file) index.html
(file) toto-1.0.0.whl
-> (file) index.html
the index.html files look somehow normal (same as in the pep-503). I try, from another computer, say host2 that has access to this shared directory to install toto package using:
pip install --index-url file://host1/my-pipy toto
but it fails (when it tries to read the file with a OSError: [Errno 22] Invalid argument: '\\\\host1\\my-pypi\\').
First, has anyone tried (and solved) this before? (That's the easy solution :-) ).
Second, I've dug a little in the code of pip, and there are a couple of things that are unclear to me (if there's ever a pro in pip that can answer ;-) ).
index.py, method find_all_candidates: it automatically changes my url list from [file://host1/my-pypi/toto], so it seems that it never tries to read host1\my-pypi\index.html... weird?
index.py, method get_page: well point 1 is not blocking for me as it magically matches my architecture, but there's a weird condition:
# Tack index.html onto file:// URLs that point to directories
(scheme, netloc, path, params, query, fragment) = \
urllib_parse.urlparse(url)
if (scheme == 'file' and
os.path.isdir(urllib_request.url2pathname(path))):
# add trailing slash if not present so urljoin doesn't trim
# final segment
if not url.endswith('/'):
url += '/'
url = urllib_parse.urljoin(url, 'index.html')
logger.debug(' file: URL is directory, getting %s', url)
resp = session.get(
url,
headers={
"Accept": "text/html",
"Cache-Control": "max-age=600",
},
)
well, as expected, we have:
scheme = 'file'
netloc = 'host1'
path = '/my-pypi/'
but the check condition os.path.isdir(urllib_request.url2pathname(path))) is obviously false as we've discarded the network location. Hence index.html is not appended to the path (that would've been correct otherwise), and hence the error while trying to read a file that does not exist.
For a remote UNC file share, specify the file URL with four slashes:
pip install --index-url file:////host1/my-pypi toto
In this case, the parsed netloc is empty and the path still includes the server hostname in UNC format, so isdir works correctly and index.html will be appended as expected.
Related
When using:
#route('/<filename>')
def server_static(filename):
return static_file(filename, root='.')
it allows to serve the request www.example.com/helloworld with the static HTML file /myapp/helloworld.
How to make www.example.com/anything be served by the static HTML file /myapp/html/anything.html, without having to hardcode each static filename anything in the Python code?
Note: the tricky part is that the request is /anything (and not /anything.html), and the static file is /myapp/html/anything.html (there are 20 or more such files)
If it's always .html you could consider simply doing filename += '.html'.
If your needs are more complex, you could write code that examines the directory to match various extensions. For example, if you wanted to match .html files, something like this would work and would be adjustable to work with various/multiple extensions or other conditions.
def is_html(filename):
return filename.lower().endswith('.html')
#route('/<filename>')
def server_static(filename):
root = '.'
all_filenames = os.listdir(root)
html_files = filter(is_html, all_filenames)
for fname in html_files:
if fname.startswith(filename):
return static_file(fname, root=root)
else:
return "No such file"
This can probably be abbreviated using fnmatch or something similar.
Although, you may want to simply consider simply having a webserver like Apache simply serve that path instead of using Flask. This would probably be a more secure option, too.
So I am trying to create a bot that cross posts from a sub (r/pics) to (r/polpics) using a bit of code from u/GoldenSights. I upgraded to a new python distro and I get a ton of errors, I don't even know where to begin. Here is the code (formatting off, error lines bold):
Traceback (most recent call last):
File "C:\Users\tonyc\AppData\Local\Programs\Python\Python36-32\Lib\site-
packages\praw\subdump.py", line 84, in <module>
r = praw.Reddit(USERAGENT)
File "C:\Users\tonyc\AppData\Local\Programs\Python\Python36-32\lib\site-
packages\praw\reddit.py", line 150, in __init__
raise ClientException(required_message.format(attribute))
praw.exceptions.ClientException: Required configuration setting 'client_id'
missing.
This setting can be provided in a praw.ini file, as a keyword argument to the `Reddit` class constructor, or as an environment variable.
This seems to be related to USERAGENT setting. I don't think I have that configured right.
USERAGENT = ""
# This is a short description of what the bot does. For example
"/u/GoldenSights' Newsletter bot"
SUBREDDIT = "pics"
# This is the sub or list of subs to scan for new posts.
# For a single sub, use "sub1".
# For multiple subs, use "sub1+sub2+sub3+...".
# For all use "all"
KEYWORDS = ["It looks like this post is about US Politics."]
# Any comment containing these words will be saved.
KEYDOMAINS = []
# If non-empty, linkposts must have these strings in their URL
This is the error line:
print('Logging in')
r = praw.Reddit(USERAGENT) <--here, this is error line 84
r.set_oauth_app_info(APP_ID, APP_SECRET, APP_URI)
r.refresh_access_information(APP_REFRESH)
Also in Reddit.py :
raise ClientException(required_message.format(attribute)) <--- error
praw.exceptions.ClientException: Required configuration setting 'client_id'
missing.
This setting can be provided in a praw.ini file, as a keyword argument to
the `Reddit` class constructor, or as an environment variable.
Firstly, you're going to want to have your API credentials stored externally in your praw.ini file. This makes things a lot more secure, and looks like it might go some way to fixing your issue. Here's what a completed praw.ini file looks like, including the useragent, so try to replicate this.
[DEFAULT]
# A boolean to indicate whether or not to check for package updates.
check_for_updates=True
# Object to kind mappings
comment_kind=t1
message_kind=t4
redditor_kind=t2
submission_kind=t3
subreddit_kind=t5
# The URL prefix for OAuth-related requests.
oauth_url=https://oauth.reddit.com
# The URL prefix for regular requests.
reddit_url=https://www.reddit.com
# The URL prefix for short URLs.
short_url=https://redd.it
[appname]
client_id=IE*******T14_w
client_secret=SW***********************CLY
password=******************
username=appname
user_agent=web:appname:1.0.0 (by /u/username)
Let me know how things go after you sort this out.
I'm using django-pipeline with s3. I'm successfully using collectstatic to combined my Javascript files and store them in my s3 bucket, but they are not getting compressed for some reason (verified by looking at the file, its size, and its content-encoding). Otherwise things are working correctly with the combined scripts.js that is produced.
Here are the changes I made to use django-pipeline:
Added pipeline to installed apps.
Added 'pipeline.finders.PipelineFinder' to STATICFILES_FINDERS.
Set STATICFILES_STORAGE = 'mysite.custom_storages.S3PipelineManifestStorage' where this class is as defined in the documentation, as seen below.
Set PIPELINE_JS as seen below, which works but just isn't compressed.
PIPELINE_ENABLED = True since DEBUG = True and I'm running locally.
PIPELINE_JS_COMPRESSOR = 'pipeline.compressors.yuglify.YuglifyCompressor' even though this should be default.
Installed the Yuglify Compressor with npm -g install yuglify.
PIPELINE_YUGLIFY_BINARY = '/usr/local/bin/yuglify' even though the default with env should work.
Using the {% load pipeline %} and {% javascript 'scripts' %} which work.
More detail:
PIPELINE_JS = {
'scripts': {
'source_filenames': (
'lib/jquery-1.11.1.min.js',
...
),
'output_filename': 'lib/scripts.js',
}
}
class S3PipelineManifestStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
location = settings.STATICFILES_LOCATION
As mentioned, collectstatic does produce scripts.js just not compressed. The output of that command includes:
Post-processed 'lib/scripts.js' as 'lib/scripts.js'
I'm using Django 1.8, django-pipeline 1.5.2, and django-storages 1.1.8.
Similar questions:
django-pipeline not compressing
django pipeline with S3 storage not compressing
The missing step was to also extend GZipMixin, AND, it has to be first in the list of parents:
from pipeline.storage import GZIPMixin
class S3PipelineManifestStorage(GZIPMixin, PipelineMixin, ManifestFilesMixin, S3BotoStorage):
location = settings.STATICFILES_LOCATION
Now collectstatic produces a .gz version of each file as well, but my templates still weren't referencing the .gz version.
To address this the author says:
To make it work with S3, you would need to change the staticfiles
storage url method to return .gz urls (and staticfiles/pipeline
template tags depending if you care for clients that don't support
gzip). Also don't forget to setup the proper header on s3 to serve
theses assets as being gzipped.
I adapted an example he provided elsewhere, which overrides the url method:
class S3PipelineManifestStorage(GZIPMixin, PipelineMixin, ManifestFilesMixin, S3BotoStorage):
location = settings.STATICFILES_LOCATION
def url(self, name, force=False):
# Add *.css if you are compressing those as well.
gzip_patterns = ("*.js",)
url = super(GZIPMixin, self).url(name, force)
if matches_patterns(name, gzip_patterns):
return "{0}.gz".format(url)
return url
This still doesn't handle setting the Content-Encoding header.
A simpler alternative is to use the S3Boto Storages option AWS_IS_GZIPPED which performs gzipping AND sets the appropriate header.
More is required to support clients without gzip, however.
Also useful are these instructions from Amazon on serving compressed files from S3.
I am going through a short Python tutorial, but I can't get the last exercise to work.
This is the source code of app.py
import web
urls = (
'/', 'Index'
)
app = web.application(urls, globals())
render = web.template.render('templates/')
class Index(object):
def GET(self):
greeting = "Hello World"
return render.index(greeting = greeting)
if __name__ == "__main__":
app.run()
and this is the view, index.html
$def with (greeting)
<html>
<head>
<title>Gothons of Planet Percal #25</title>
</head>
<body>
$if greeting:
I just wanted to say <em style="color: green; font-size: 2em;">
greeting</em>.
$else:
<em>Hello</em>, world!
</body>
</html>
The file app.py is under this directory: C:\Users\Lucas\Desktop\Learn Python The Hard Way\ex50\gothonweb\bin
and index.html is at: C:\Users\Lucas\Desktop\Learn Python The Hard Way\ex50\gothonweb\templates
So, when I want to run the sample code, I type this in the command prompt:
C:\Python26\python.exe "C:\Users\Lucas\Desktop\Learn Python The Hard Way\ex50\gothonweb\bin\app.py"
After that, "http://0.0.0:8080" is displayed on the console, so I go to http://localhost:8080/ in my browser
but I get back a long traceback starting with
<type 'exceptions.AttributeError'> at /
No template named index
Python C:\Python26\lib\site-packages\web\template.py in _load_template, line 992
Web GET http://localhost:8080/
What is going on and how do I fix it?
Thanks in advance!
I had this problem as well but running on OSX. Ultimately Zed Shaw saw my pleas for help and saw the mistake I was making.
I was running
~/projects/gothonweb/bin $ python app.py
Zed reminded me that I needed to be running this:
~/projects/gothonweb $ python bin/app.py
to allow the templates folder to be found. After I did that it worked like a charm.
in windows ,the folder'name must write like this "c:\" not "c/",and you must use full path.
so the right code is render = web.template.render('d:\\documents\\python\\templates\\')
(app.py is in d:\documents\python)
You have a few typos, you need to refer to your view as Index when you use render (needs to be the same as the class name for your route):
return render.Index(greeting = greeting)
And your urls tuple needs a trailing comma:
urls = (
'/', 'Index',
)
Also make sure your template is named Index.html. Although, looking at the web.py tutorial, it looks like by convention you'd use lowercase for your route class.
Well, I suffered from the same problem, and I must say the error message is right, which indicates "you" cannot find the file, simply because you are not in the right path. So #davidheller #shellfly are right.
I use PyCharm as IDE to write python, so here is my solution:
Since I run the app.py, which is under the bin directory, thus render = web.template.render('../templates/')
which .. goes up and then found the file.
To conclude, we must be sure about the current path(even in windows), and both relative path or absolute path can be used in Windows environment, as shown below:
Absolute path.
Absolute path. Since Windows accepts both "/" and "\", we can write
render = web.template.render('C:/Users/Lucas/Desktop/Learn Python The Hard Way/ex50/gothonweb/templates/')
or
render = web.template.render('C:\\Users\\Lucas\\Desktop\\Learn Python The Hard Way\\ex50\\gothonweb\\templates\\')
Note, python interpreter interprets "\\" as "\"
Relative path.
render = web.template.render('../templates/')
You may need to compile the template like so
python web/template.py --compile templates
anyone using web.py with the Google app engine you will need to.
I am doing the same exercise and I simply go on cmd , cd to my lpthw directory which contains the folders of the project skeleton inside and do:
> python bin/app.py
I think you have to put all your files from the project skeleton in one folder and run your app from there. Hope this helps. :)
We have two zope servers running our company's internal site. One is the live site and one is the dev site. I'm working on writing a python script that moves everything from the dev server to the live server. Right now the process involves a bunch of steps that are done in the zope management interface. I need to make all that automatic so that running one script handles it all. One thing I need to do is export one folder from the live server so that I can reimport it back into the live site after the update. How can I do this from a python script?
We're using Zope 2.8 and python 2.3.4
You can try to use the functions manage_exportObject and manage_importObject located in the file $ZOPE_HOME/lib/python/OFS/ObjectManager.py
Let say we install two Zope 2.8 instances located at:
/tmp/instance/dev for the development server (port 8080)
/tmp/instance/prod for the production server (port 9090)
In the ZMI of the development server, I have created two folders /MyFolder1 and /MyFolder2 containing some page templates. The following Python script exports each folder in .zexp files, and imports them in the ZMI of the production instance:
#!/usr/bin/python
import urllib
import shutil
ids_to_transfer = ['MyFolder1', 'MyFolder2']
for id in ids_to_transfer:
urllib.urlopen('http://admin:password_dev#localhost:8080/manage_exportObject?id=' + id)
shutil.move('/tmp/instance/dev/var/' + id + '.zexp', '/tmp/instance/prod/import/' + id + '.zexp')
urllib.urlopen('http://admin:password_prod#localhost:9090/manage_delObjects?ids=' + id)
urllib.urlopen('http://admin:password_prod#localhost:9090/manage_importObject?file=' + id + '.zexp')
To make this more general and allow copying folders not in the root directory I would do something like this:
#!/usr/bin/python
import urllib
import shutil
username_dev = 'admin'
username_prod = 'admin'
password_dev = 'password_dev'
password_prod = 'password_prod'
url_dev = 'localhost:8080'
url_prod = 'localhost:9090'
paths_and_ids_to_transfer = [('level1/level2/','MyFolder1'), ('level1/','MyFolder2')]
for path, id in ids_to_transfer:
urllib.urlopen('http://%s:%s#%s/%smanage_exportObject?id=%s' % (username_dev, password_dev, url_dev, path, id))
shutil.move('/tmp/instance/dev/var/' + id + '.zexp', '/tmp/instance/prod/import/' + id + '.zexp')
urllib.urlopen('http://%s:%s#%s/%smanage_delObjects?ids=%s' % (username_prod, password_prod, url_prod, path, id))
urllib.urlopen('http://%s:%s#%s/%smanage_importObject?file=%s.zexp' % (username_prod, password_prod, url_prod, path, id))
If I had the rep I would add this to the other answer but alas...
If someone wants to merge them, please go ahead.
If you really move everything you could probably just move the Data.fs instead. But otherwise the import/export above is a good way.