This is a rather conceptual question than a line of code question.
I'm trying to proxy an android app in order to inspect its API calls, without having root access to the device.
I'm using mitmproxy, and I've managed to create a system wide proxy using this, but I'm quite clueless as to how can I filter the wanted requests from just the given app.
I can tell the bundle id of the app, but how can I target just this app's requests? With root, this can be easy - using iptables, but how can it be done without root?
This app can be third-party, so using heuristics based on domain or header can be a bit problematic since I don't control them. Also, I would actually prefer a solution style iptables, but without root.
Thanks!
Related
Background: Am comfortable in Python, know nothing about web deployment. I am looking into it as an alternative to compiling into .exe or .app for Win or Mac distributions.
Issue: I have a simple application that uses BeautifulSoup, openpyxl, and PySimplyGUI. It interacts with some local excel-files and creates new ones. I want to be able to, using minimum effort, make it accessible on my own web page or something similar, and make the created excel-files available for browsing/download. I have no idea how to do any of this. I've been looking into Flask and cloud foundry, but it feels like there should be some easy alternative that I'm missing. Ideally I would want a page where someone can log in (given a username and password I supply), which then directs to a page where the user can interact with my application.
Request: Is there a relatively easy way to do this that doesn't involve setting up a lot of stuff in html, etc., and where excel-files can still be interacted with by openpyxl? I ideally would just want some template, where I can "fill in the blanks" for the python method I would want to execute for each button!
Hope this makes sense. Thanks in advance :)
The easiest way to create a web app with a simple interface yet effective which does not require frontend programming is Streamlit. It is primarily used by data scientist to create simple web apps quickly.
Let's say I am creating a python-based CMS on GAE (similar to Squarespace/Shopify) which allows users to create a website.
The platform will (automatically?) create a subdomain for each new user and duplicates the application.
Now there are two options:
1) Create a new Database for the new user, WITHIN the master GAE project. (I'm worried that if one user gets a lot of traffic it might slow down ALL websites.)
2) Duplicate the entire project. (This method seems difficult to accomplish because either I have to manually create an instance of the application for each user, or I have to figure out how to hijack gcloud.py (or appcfg.py) somehow and store my login credentials in the code.)
Which choice will most likely provide the most performance for the price? Is choice 2 allowed by Google (or even possible)?
Edit:
I've done some more research about this, and it's not documented very much. I found this in the docs https://cloud.google.com/sdk/docs/scripting-gcloud which talks about running gcloud from scripts, although I don't think that means from python. I am looking into appengine-jenkins to see if it will work for my purpose. Let me know if you have any additional information about this.
Also, it seems like gcloud is adding a create command within the projects command which might be useful for me if I can figure out how to run gcloud from my script. https://cloud.google.com/sdk/gcloud/reference/alpha/projects/create
I have zero experience with website development but am working on a group project and am wondering whether it would be possible to create an interaction between a simple html/css website and my python function.
Required functionality:
I have to take in a simple string input from a text box in the website, pass it into my python function which gives me a single list of strings as output. This list of strings is then passed back to the website. I would just like a basic tutorial website to achieve this. Please do not give me a link to the CGI python website as I have already read it and would like a more basic and descriptive view as to how this is done. I would really appreciate your help.
First you will need to understand HTTP. It is a text based protocol.
I assume by "web site" you mean User-Agent, like FireFox.
Now, your talking about an input box, well this will mean that you've already handled an HTTP request for your content. In most web applications this would have been several requests (one for the dynamically generated application HTML, and more for the static css and js files).
CGI is the most basic way to programmatically inspect already parsed HTTP requests and create HTTP responses from objects you've set.
Now your application is simple enough where you can probably do all the HTTP parsing yourself to gain a basic understanding of what's going on, but you will still need to understand how to develop a server that can listen on a socket.
To avoid all that just find a Python application server that has already implemented all of the above and much more. There are many python application servers to choose from. Use one with a small learning curve for something as simple as above. Some are labeled as "micro-frameworks" in this genre.
Have you considered creating an app on Google App Engine (https://developers.google.com/appengine/)?
Their Handling Forms tutorial seems to describe your problem:
https://developers.google.com/appengine/docs/python/gettingstartedpython27/handlingforms
I am looking for a lib that lets me roughly:
connect to localhost:port, but see http://somesite.com
rewrite all static assets to point to localhost:port instead of somesite.com
support cookies / authentication
i know that http://betterinternet.co/ does this already, but they wont give me their source code for some reason.
I assume this doesnt exist as free code, so if i were to write one, are there any nuances to it? If i replace all occurrences of somesite.com in html and headers, will that be enough?
So...you want an http proxy that does link rewriting? Sounds like Apache and mod_proxy_html. It's not written in node or Python, but I think it will do what you're asking.
I don't see any straight forward solution to your problem. If I've understood correctly, you want a caching HTTP proxy which serves static contents locally, with URL rewriting rules defined in Python (or nodejs). That's quite a task.
A caching HTTP proxy implementation is not trivial. So I'd use an existing implementation, such as Squid (or Apache if it does caching too).
You could then place a (relatively) simple HTTP server written in Python in front of that (e.g. based on BaseHTTPServer and urllib2) which performs the URL rewriting as you want them and forwards the requests to the proxy (or direct to internet).
The idea would be to rely on the proxy setup to perform all the processing you don't want to modify (including basic rewrite rules, authentication, caching and cache management) and limit your front-end implementation to performing only the custom rewriting you are interested in.
I'm planning an iOS app that requires a server backend capable of efficiently serving image files and performing some dynamic operations based on the requests it gets (like reading and writing into a data store, such as Redis). I'm most comfortable with, and would thus prefer to write the backend in Python.
I've looked at a lot of Python web framework/server options, Flask, Bottle, static and Tornado among them. The common thread seems to be that either they support serving static files as a development-time convenience only, discouraging it in production, or are efficient static file servers but not really geared towards the dynamic framework-like side of things. This is not to say they couldn't function as the backend, but at a quick glance they all seem a bit awkward at it.
In short, I need a web framework that specializes in serving JPEGs instead of generating HTML. I'm pretty certain no such thing exists, but right now I'm hoping that someone could suggest a solution that works without bending the used Python applications in ways they are not meant for.
Specifications and practical requirements
The images I'd be serving to the clients live in the file system in a shallow directory hierarchy. The actual file names would be invisible to the clients. The server would essentially read the directory hierarchy at startup, assigning a numeric ID for each file, and would route the requests to controller methods that then actually serve the image files. Here are a few examples of ways the client would want to access the images in different circumstances:
Randomly (example URL path: /image/random)
Randomly, each file only once (/image/random_unique), produces some suitable non-200 HTTP status code when the files are exhausted
Sequentially in either direction (/image/0, /image/1, /image/2 etc.)
and so on. In addition, there would be URL endpoints for things like ratings, image info and other metadata, some client-specific information as well (the client would "register" with the server, so that needs some logic, too). This data would live in a Redis datastore, most likely.
All in all, the backend needs to be good at serving image/jpeg and application/json (which it would also generate). The scalability and concurrency requirements are modest, at least to start with (this is not an App Store app, going for ad-hoc or enterprise distribution).
I don't want the app to rely on redirects. That is, I don't want a model where a request to a URL would return a redirect to another URL that is backed by, say, nginx as a separate static file server, leaving only the image selection logic for the Python backend. Instead, a request to a URL from the client should always return image/jpeg, with metadata in custom HTTP headers where necessary. I specify this because it is a way of avoiding serving static files from Python that I thought of, and someone else might think of too ;-)
Given this information, what sort of solution would you consider a good choice, and why? Or is this something for which I need to code non-trivial extensions to existing projects?
EDIT: I've been thinking about this a bit more. I don't want redirects due to the delay inherent in the multiple requests they entail, plus I'd like to abstract out the file names from the client, but I was wondering if something like this would be possible:
It's pretty self-explanatory, but the idea is that the Python program is given the request info by nginx (or whatever serves the role), mulls it over and then tells nginx to respond to the client's request with a specific file from the file system. It does so. The client is none the wiser about how the request was fulfilled, it just receives a response with the correct content type.
This would be pretty optimal in my view, but is it possible? If not with nginx, perhaps something else?
I've been using Django for well over a year now, and it is the hammer I use for all my nails. You could probably do this with a bit of database-image storage and django's builtin orm and url routing (with regex). If you store the images in the database, you will automatically get the unique-id's set. According to this stackoverflow answer, you can use redis with django.
I don't want a model where a request to a URL would return a redirect to another URL that is backed by, say, nginx as a separate static file server, leaving only the image selection logic for the Python backend.
I think Nginx for serving static and python for figuring out the image url is the better solution.
Still if you do not want to do that I would suggest you use any Python web framework (like Django) and write your models and convert them into REST resources (Eg. Using django-tastypie) and/or return a base64 encoded image which you can then decode in your iOS client.
Refs:
Decoding a Base64 image
TastyPie returns the path as default, you might have to do extra work to either store the image blob in the table or write more code to return a base64 encoded image string
You might want to look at one of the async servers like Tornado or Twisted.