I want to query the Google Analytics API using Python to periodically download data from my Analytics account and store data in a local database. I am basically following the steps as given in the basic tutorial. I am using the Google client API library for Python in this process.
My script is working fine so far when I am running it on my local dev machine (Mac). When I start the script, my browser opens and I am prompted to grant access to my Analytics data from the app. Afterwards I can run my script as often as I want and get access to my data.
On my server (Ubuntu, only terminal available), the w3m browser opens, but I cannot access my Google account from there. I can only quit w3m and kill the program with Ctrl-C. There is an error message like:
Your browser has been opened to visit:
https://accounts.google.com/o/oauth2/auth?scope=some_long_url&access_type=offline
If your browser is on a different machine then exit and re-run this
application with the command-line parameter
--noauth_local_webserver
However when I run my script with the parameter --noauth_local_webserver, I get the same results - w3m opens and I cannot authenticate.
How can I get the --noauth_local_webserver to work? I there another way to authenticate without a local browser on the same machine?
When you use FLAGS = gflags.FLAGS, you actually need to pass the command-line arguments to FLAGS (this may or may not have tripped me up as well :) ). See here for an Analytics-centric example of how to do it (code below as links tend to go away after a while). General idea is that argv arguments are passed into the FLAGS variable, which then become available to other modules.
# From samples/analytics/sample_utils.py in the google-api-python-client source
def process_flags(argv):
"""Uses the command-line flags to set the logging level.
Args:
argv: List of command line arguments passed to the python script.
"""
# Let the gflags module process the command-line arguments.
try:
argv = FLAGS(argv)
except gflags.FlagsError, e:
print '%s\nUsage: %s ARGS\n%s' % (e, argv[0], FLAGS)
sys.exit(1)
# Set the logging according to the command-line flag.
logging.getLogger().setLevel(getattr(logging, FLAGS.logging_level))
Also, turns out that we aren't alone! You can track this bug to see when this will get added the documentation.
you can also use GA as a service API:https://developers.google.com/analytics/devguides/reporting/core/v3/quickstart/service-py
this works perfectly fine. Just remmeber to convert the p12 to an unencryptet PEM file using openssl
$openssl pkcs12 -in client_secrets.p12 -nodes -nocerts > client_secrets.pem
the import password is printed out when you download the P12 from google developer's console
I ran into the same issue and managed to solve it by SSHing into my server. Example:
ssh -L 8080:127.0.0.1:8080 <server-name>
I then ran my script through SSH. When I was presented with the URL (https://accounts.google.com/o/oauth2/auth?scope=some_long_url&access_type=offline), I copied and pasted into the browser on my machine to complete the authentication flow.
I ran it on my PC, got a token.json, and just copied the token on the server in the home folder (think working directory of the script), it solved it.
No authentication needed if you use same token
Related
On My server, i have installed spleeter which is a python app. I am able to successfully call the spleeter commands in my terminal to execute my required commands.
I want to be able to execute those terminal commands with an http get request.
Basically, i have an android app thats supposed to make a Get request to my vps server which executes the spleeter commands and returns the response.
Please how can i create the Get endpoint. I need pointer please.
So, in the most simplistic sense, you can create a python web app (using Django, Flask or raw Python, etc.) that serves an HTTP endpoint. In that web app, you can do something like this that will run commands inside a python function, depending on the contents of the GET request:
import os
def run_command(command):
if command = "ls":
os.system('ls -l')
...
The only problem here is that, without authentication, this is horribly insecure. Anyone with access to the internet could call your endpoint and run those commands, unless you add some sort of authentication into your Android and/or Python app to validate requests.
Even passing a simple "token" in a custom HTTP header from the app to the web server might be better than nothing to verify it is a legit user sending the command.
The other problem here is that the Python (or web server) user has to have permissions to run the command, that is another security issue altogether.
Alternatively, you could have the endpoint store the commands it received via GET requests into a Redis queue or something, and then have a separate script running locally validate the commands and run them on the system so at least you separate the web portion from the server user portion.
Without more details, it is hard to know exactly what you want to accomplish.
Reference: https://janakiev.com/blog/python-shell-commands/
I have banged my head against the wall on this for days now, I need help.
I’m trying to create a geoprocessing service (which will be used via a flex viewer) that takes permit information from a GDB, writes it to a file, and then opens it on the user’s computer.
The script I have written works perfectly on local computers and when run from the server itself.
However, as soon as the script has been published as a geoprocessing service, it stops opening the file.
The file is created out on a webserver, and then opened via url so that the user can then make a local copy, print out for signature, etc. The file is still being created, named, and placed in the proper folder, it just isn’t being opened at end of the process. (The GP Service views as completed though, it doesn’t fail)
Here is the end of the code, which is the part relevant to my issue:
f = open(r'\\MyWebAddressHere\\Reports\\Permit' + str(PermitNumber) +'.html','w')
f.write(permitStart)
if (BasePermitFee > 0):
f.write(permitBaseFee)
if (PlumbingFee > 0):
f.write(permitPlumbingFee)
if (ElectricFee > 0):
f.write(permitElectricFee)
if (TotalFees > 0):
f.write(permitTotalFee)
f.write(permitEnd)
f.close()
webbrowser.open('MyWebAddressHere/Reports/Permit' + str(PermitNumber) +'.html')
Does anyone have a suggestion on why the open is failing once the GP service is published?
I don't see this as a bug - the webbrowser command makes no sense in the context of a GP Service. Think about what's being executed where.
If you run the code locally, Python is running on your actual computer - that computer has a login, a display, a set of installed programs. Telling it to open a web browser makes sense because you have one or more browsers installed on that computer.
Run it on the server, while logged into the server, and same thing - you still have a login, a set of installed programs. While you might lock down a server to reduce its vulnerability, it probably still has at least one program installed that is capable of browsing the internet.
But when you run a script as a GP Tool, you're in a different world. You're not running the code locally, and you're not logged into the server. If you execute the webbrowser command, where should it open the browser window?
On your PC? It can't - the server's GP environment can't execute random programs on your PC, that would be a massive security hole. Hey, I'll write a script that deletes everything in C:\Windows and publish it on ArcGIS Online!
On the server? Who would see it? No one's logged in.
Turns out that the webbrowser command flat out does not work in a server environment.
I managed to confuse the heck out of an ESRI server analyst with this one, because there is apparently no documentation saying this won't work in a server environment, it just doesn't. So it has been logged as a bug.
I have large video files (~100GB) that are local on my machine. I have a non-local website where I enter information about the video file. In addition, I need to get the checksum of the video file (and I do not want to manually trigger the script locally and copy and paste the value). To get a checksum of the video file, I have a script I can run as $ checksum.py <video file>.
How would I trigger the local script through the web interface? In other words, I want to be able to enter the PATH of the video file and click Submit on the web app, and it will locally trigger the script, and (after the checksum has finished), insert that value into the web app/database. What would be the best way to do this?
You cannot trigger anything unless your local script is continuously listening for some kind of data feed (like a fixed URL serving an XML/JSON feed of paths) which is, IMHO, over-complicating your system.
You could also use a Java applet ran locally instead of a remote website, but you'll have to sign it to be able to read local files, and it might not be what you're looking for.
Think of it: it's all about security. Would you like any web server to trigger scripts in your local machine? I certainly wouldn't.
IMHO the best solution is to trigger the script manually which will send the data to your web server.
In general browsers run in a sandbox that has very limited access to the OS. In particular you can't run shell scripts from a browser. As I see it you have two options:
Adapt your checksum.py script to send the checksum info directly to your website using the urllib2 calls, or pipe the results to a "curl" command. (No browser involved.)
Rewrite checksum.py as JavaScript using the FileReader class. This will probably be convoluted, slow, and won't work in Internet Explorer.
I am trying to install and start a simple CherryPy server as a Windows service.
Here is the script: (Removed some lines to cut it short. It's fully working when executing manually from the command-line)
app = AdminMediaHandler(django.core.handlers.wsgi.WSGIHandler())
logged_app = TransLogger(app)
server = wsgiserver.CherryPyWSGIServer( ('127.0.0.1', 8632), logged_app, server_name='localhost', numthreads=20 )
try:
server.start()
except KeyboardInterrupt:
server.stop()
I'm using sc.exe to install and start the service. Installation goes fine, but I can't seem to start the service.
The command used is: (note there're spaces in the paths, though I'm handeling this with double-quotes, and the binPath is working when executing its string manually through the command-line)
> sc.exe create "ServiceName" binPath= "\"C:\Path to Python\python.exe\" \"C:\Path to CherryPy Script\cherryserver.py\""
> sc.exe start "ServiceName"
I keep getting this error, no matter if attempting to start the service using sc.exe or through services.msc GUI:
[SC] StartService FAILED 1053:
The service did not respond to the start or control request in a timely fashion.
From what I understand, this is happenning because python.exe doesn't implement the Windows Service API.
I do not wish to create an .exe from the script, using py2exe.
I have found this answer that suggests to install the service using different tools than sc.exe, called srvany.exe & instsrv.exe. However, I can't find them in the Win2K Resource Kit website.
Does anybody know how to install & start this .py as a Windows succesfully?
Does anybody know
CherryPy ships with a module for starting as a Windows service. See this other SO question for instructions on how to install and run it. You'll probably want to switch from your current approach (of passing the Django app directly to the WSGIServer) and use cherrypy.tree.graft instead.
I prefer nssm for installing normal scripts as a service.
You can copy the nssm.exe in the C:\Windows\system32 or C:\Windows\SysWOW64 directory depending on your system. After that you are able to install a Service as follow:
nssm install yourservicename
For a python script you must set the application path to your python.exe and the argument is your script self.
Other common commands for start/stop/edit your service are:
nssm start yourservicename
nssm stop yourservicename
nssm edit yourservicename
I eventually used ServiceInstaller aka SMaster, as stated in this answer. The URL in the given answer is broken, and I couldn't find a working URL. I just had srunner.exe available locally beforehand.
Note there was another obstacle to overcome though, as ServiceInstaller can't handle files with spaces in their paths.
So, I used the old DOS path formatting for service registration.
Instead of registering C:\Program Files\MyApp\python.exe, I registered C:\PROGRA~1\MyApp\python.exe.
I'm looking to use a local webserver to run a series of python scripts for the user. For various unavoidable reasons, the python script must run locally, not on a server. As a result, I'll be using HTML+browser as the UI, which I'm comfortable with, for the front end.
I've been looking, therefore, for a lightweight web server that can execute python scripts, sitting in the background on a machine, ideally as a Windows service. Security and extensibility are not high priorities as it's all running internally on a small network.
Should I run a native python webserver as a Windows service (in which case, how)? Or is it just as easy to install Apache onto the user's machine and run as CGI? Since this is all local, performance is not an issue either.
Or am I missing something obvious?
Don't waste a lot of time creating Windows service.
Don't waste a lot of time on Windows Apache.
Just make a Python service that responds to HTTP requests.
Look at https://docs.python.org/2/library/basehttpserver.html
https://docs.python.org/3/library/http.server.html for version 3
Python offers an HTTP server that you can extend with your server-side methods.
Look at http://docs.python.org/library/wsgiref.html
Python offers a WSGI reference implementation that makes your server easy and standards-compliant.
Also http://fragments.turtlemeat.com/pythonwebserver.php
"I'm trying to avoid making the user run python stuff from the command prompt."
I don't see how clicking a web page is any different from clicking desktop icons.
Starting a web server based on Python is relatively easy, once you have the web server. First, build the server. Later, you can make sure the server starts. Let's look at some ways.
Your user can't use a random browser to open your local page. They need a bookmark to launch "localhost:8000/myspecialserverinsteadofthedestop/" That bookmark can be a .BAT file that (1) runs the server, (2) runs firefox with the proper initial URL.
You can put the server in the user's start-this menu.
You can make your Python program a windows "service".
Best way is to make your own local server by using command prompt.
Make a new folder say Project
Make a new folder inside project & name it as "cgi-bin"(without quotes)
Paste your .py file inside the cgi-bin folder
Open cmd and change to the directory from which you want to run the server and type "python -m CGIHTTPServer"(without quotes)
Minimize the cmd window & open your browser and type "localhost:8000/cgi-bin/yourpythonfilename.py"(without quotes).
The wasiest step would be navigate to folder where your files are located and running http.server module
cd /yourapp
python3 -m http.server
the you should see something like this in console
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
Running a native python webserver as a windows service should be a no brainer. Check out the documentation for writing windows services (win32api, ActiveState python) in python and also the documentation for subclassing BaseHttpServer and SimpleHttpServer.
BTW: I had a similar question on stackoverflow: How to stop BaseHTTPServer.serve_forever() in a BaseHTTPRequestHandler subclass?
Basically, you subclass BaseHTTPServer (you have to anyway...) and then... but just read the accepted answer - it set me on the right track!