Develop Reference

Python - Different logs should be in different files, but appear on the same file

I have a web application, with a minimal logging functionality. The backend runs on Apache with Falcon (on Python).
Each request has the following code
msg = 'user: {usr} running {req} {req_uri:<30} | from: {loc_ip}:{loc_port} '.format(
usr=req.env['REMOTE_USER'],
req=req.env['REQUEST_METHOD'],
loc_ip=req.env['REMOTE_ADDR'],
loc_port=req.env['REMOTE_PORT'],
req_uri=req.env['REQUEST_URI'])
log_name = 'logs/{remote_ip}/{remote_ip}_{day}.log'.format(remote_ip=req.env['REMOTE_ADDR'], day=datetime.datetime.now().date().strftime('%d_%m_%Y'))
os.makedirs(os.path.dirname(log_name), exist_ok=True)
logging.basicConfig(filename=log_name, level=logging.DEBUG, format='[%(asctime)s] - %(levelname)s - [%(module)s:%(lineno)d] %(message)s', datefmt='%d/%m/%Y %H:%M:%S')
logging.info(msg)
And I access the app from 2 different IPs (IPx, and IPy) - so there should be 2 folder under 'logs', folder 'IPx' and folder 'IPy', with a log file in each..
But after I access the web, I see 2 folders, but only one of the folders has a log file inside (lets say IPx_day.log in IPx folder), but after reviewing the IPx_day.log file I see:
[TIME] - INFO - [FILE] user: user1 running GET /domain | from: IPy:PORTy
// Other log statments..
[TIME] - INFO - [FILE] user: user2 running GET /domain | from: IPx:PORTx
both lines are in the file file - IPx_day.log
Has anyone have idea why both lines appear in the same file?
Thanks.

[I would add this as a comment but I don't have the reputation to do so.]
I think it might have something to do with logging.basicConfig() which only configures the root logger. I think you'd have to use multiple loggers or better just attach multiple handlers to one logger.
https://docs.python.org/3/library/logging.html#logging.basicConfig
https://docs.python.org/3/library/logging.handlers.html

From your writing, what I have understood is
You're creating 2 different folders and the log file in respective
directories.
You run a app from IPx, and you see logs in IPx.log but
when you run app from IPy, logs for IPy are in IPx.log.
If that is right, can you please check if
Right IP[remote_IP] is being passed all the time?
Print the IP and log_name and check what IP you're getting and the the log directory.
I presume, a path change is required. Unless you're in the base path[source of the script]

Not Found Error (404) with Static files when running from prompt

I recently moved from windows to raspberry pi for my app. It loaded at least once but now for the life of me I can't get static files to load again.
If I run the python script from shell as sudo (or without) I get 404 for all static files, dynamic links still work as expected.
If I run it from IDLE logging in as 'pi' it works fine.
Relevant code:
from bottle import route, run, get, request, static_file
#get('/pumps')
def pumpData():
return json.dumps(pump.getPumps())
# root dir
#route('/<filename>')
def server_static(filename):
return static_file(filename, root='')
# css dir
#route('/css/<filename>')
def server_static(filename):
return static_file(filename, root='css')
run(host='myip', port=2000, debug=True)
What could be causing the issue? I could guess its something to do with permissions but I dont know how I would fix it.

I don't think it's a permission problem. (That would return a 403.) It's most likely a path issue.
The good news is: fixing it should be straightforward. (Famous last words. ;) You should either
specify absolute an path as the root param to static_file, or
call os.chdir() into the static file root before you call bottle.run.
So, this:
return static_file(filename, root='/path/to/your/static/file/root')
or this:
os.chdir('/path/to/your/static/file/root')
run(host='myip', port=2000, debug=True)

cannot access webserver resources using virtualenv and webapp2

I wanted to create a simple app using webapp2. Because I have Google App Engine installed, and I want to use it outside of GAE, I followed the instructions on this page: http://webapp-improved.appspot.com/tutorials/quickstart.nogae.html
This all went well, my main.py is running, it is handling requests correctly. However, I can't access resources directly.
http://localhost:8080/myimage.jpg or http://localhost:8080/mydata.json
always returns a 404 resource not found page.
It doesn't matter if I put the resources on the WebServer/Documents/ or in the folder where the virtualenv is active.
Please help! :-)
(I am on a Mac 10.6 with Python 2.7)

(Adapted from this question)
Looks like webapp2 doesn't have a static file handler; you'll have to roll your own. Here's a simple one:
import mimetypes
class StaticFileHandler(webapp2.RequestHandler):
def get(self, path):
# edit the next line to change the static files directory
abs_path = os.path.join(os.path.dirname(__file__), path)
try:
f = open(abs_path, 'r')
self.response.headers.add_header('Content-Type', mimetypes.guess_type(abs_path)[0])
self.response.out.write(f.read())
f.close()
except IOError: # file doesn't exist
self.response.set_status(404)
And in your app object, add a route for StaticFileHandler:
app = webapp2.WSGIApplication([('/', MainHandler), # or whatever it's called
(r'/static/(.+)', StaticFileHandler), # add this
# other routes
])
Now http://localhost:8080/static/mydata.json (say) will load mydata.json.
Keep in mind that this code is a potential security risk: It allows any visitors to your website to read everything in your static directory. For this reason, you should keep all your static files to a directory that doesn't contain anything you'd like to restrict access to (e.g. the source code).

Different page in different folder

I read some tutorial in Google App Engine, in those tutorial there is same structure: each page point to each class, and ALL this class always same in main.py. I want to create new page on new file. So I do:
In project folder. I create hello folder, in this I create a simple script hello.py that has a class name Hello. For example this file is:
class Hello(webapp2.RequestHandler):
def get(self):
self.response.out.write('hello world')
app = webapp2.WSGIApplication([('/hello',Hello)], debug=True)
But when I run this app, when I point to : localhost:port/hello I will receive 404:Error Resource not found..
If I define in app.yaml. I cannot deploy this app:
- url: /hello
script: hello.app
So, please teach me how to run different file in different folder. in my example is hello.py and Hello Class in this file.

Try this:
- url: /hello
script: hello/hello.py
The script field should be the complete path to the Python file that your handler lives in.

Reading files in GAE using python

I created a simple python project that serves up a couple of pages. I'm using the 'webapp' framework and django. What I'm trying to do is use one template file, and load 'content files' that contain the actual page text.
When I try to read the content files using os.open, I get the following error:
pageContent = os.open(pageUrl, 'r').read()
OSError: [Errno 1] Operation not permitted: 'content_includes/home.inc' error
If I let the django templating system to read the same file for me, everything works fine!
So the question is What am I doing wrong that django isn't??? The same 'pageUrl' is used.
The code below will give me the error, while if I comment out the first pageContent assignment, everything works fine.
Code:
pageName = "home";
pageUrl = os.path.join(os.path.normpath('content_includes'), pageName + '.inc')
pageContent = os.open(pageUrl, 'r').read()
pageContent=template.render(pageUrl, template_values, debug=True);
template_values = { 'page': pageContent,
'test': "testing my app"
}
Error:
Traceback (most recent call last):
File "/opt/apis/google_appengine/google/appengine/ext/webapp/__init__.py", line 511, in __call__
handler.get(*groups)
File "/home/odessit/Development/Python/Alpha/main.py", line 19, in get
pageContent = os.open(pageUrl, 'r').read()
File "/opt/apis/google_appengine/google/appengine/tools/dev_appserver.py", line 805, in FakeOpen
raise OSError(errno.EPERM, "Operation not permitted", filename)
OSError: [Errno 1] Operation not permitted: 'content_includes/home.inc'
app.yaml:
handlers:
- url: /javascript
static_dir: javascript
- url: /images
static_dir: images
- url: /portfolio
static_dir: portfolio
- url: /.*
script: main.py

os.path.normpath() on "content_includes" is a no-op - normpath just removes double slashes and other denormalizations. What you probably want is to build a path relative to the script, in which case you should do something like os.path.join(os.path.dirname(__file__), 'content_includes', pageName + '.inc').

If you dig in the dev_appserver.py source code and related files you see that the server does some incarnate checking to ensure that you open only files from below your applications root (in fact the rules seem even more complex).
For file access troubled I instrumented that "path permission checking" code from the development server to find that I was using absolute paths. We propbably should do a patch to appengine to provide better error reporting on that: IIRC the Appserver does not display the offending path but a mangled version of this which makes debugging difficult.