Develop Reference

Search/Filter/Select/Manipulate data from a website using Python

I'm working on a project that basically requires me to go to a website, pick a search mode (name, year, number, etc), search a name, select amongst the results those with a specific type (filtering in other words), pick the option to save those results as opposed to emailing them, pick a format to save them then download them by clicking the save button.
My question is, is there a way to do those steps using a Python program? I am only aware of extracting data and downloading pages/images, but I was wondering if there was a way to write a script that would manipulate the data, and do what a person would manually do, only for a large number of iterations.
I've thought of looking into the URL structures, and finding a way to generate for each iteration the accurate URL, but even if that works, I'm still stuck because of the "Save" button, as I can't find a link that would automatically download the data that I want, and using a function of the urllib2 library would download the page but not the actual file that I want.
Any idea on how to approach this? Any reference/tutorial would be extremely helpful, thanks!
EDIT: When I inspect the save button here is what I get:
Search Button

This would depend a lot on the website your targeting and how their search is implemented.
For some websites, like Reddit, they have an open API where you can add a .json extension to a URL and get a JSON string response as opposed to pure HTML.
For using a REST API or any JSON response, you can load it as a Python dictionary using the json module like this
import json
json_response = '{"customers":[{"name":"carlos", "age":4}, {"name":"jim", "age":5}]}'
rdict = json.loads(json_response)
def print_names(data):
for entry in data["customers"]:
print(entry["name"])
print_names(rdict)

You should take a look at the Library of Congress docs for developers. If they have an API, you'll be able to learn about how you can do search and filter through their API. This will make everything much easier than manipulating a browser through something like Selenium. If there's an API, then you could easily scale your solution up or down.
If there's no API, then you have
Use Selenium with a browser(I prefer Firefox)
Try to get as much info generated, filtered, etc. without actually having to push any buttons on that page by learning how their search engine works with GET and POST requests. For example, if you're looking for books within a range, then manually conduct this search and look at how the URL changes. If you're lucky, you'll see that your search criteria is in the URL. Using this info you can actually conduct a search by visiting that URL which means your program won't have to fill out a form and push buttons, drop-downs, etc.
If you have to use the browser through Selenium(for example, if you want to save the whole page with html, css, js files then you have to press ctrl+s then click "save" button), then you need to find libraries that allow you to manipulate the keyboard within Python. There are such libraries for Ubuntu. These libraries will allow you to press any keys on the keyboard and even do key combinations.
An example of what's possible:
I wrote a script that logs me in to a website, then navigates me to some page, downloads specific links on that page, visits every link, saves every page, avoids saving duplicate pages, and avoids getting caught(i.e. it doesn't behave like a bot by for example visiting 100 pages per minute).
The whole thing took 3-4 hours to code and it actually worked in a virtual Ubuntu machine I had running on my Mac which means while it was doing all that work I could do use my machine. If you don't use a virtual machine, then you'll either have to leave the script running and not interfere with it or make a much more robust program that IMO is not worth coding since you can just use a virtual machine.

Browser as GUI for user input to a python script

Is there an easy way to use the Chrome as a GUI to allow a user to input data for further processing in Python. What I want to do is :
The user enters data into a table. The table is 5 rows by 4 columns. The user data is then processed in JScript and a weight average of each row is displayed in column 5 of the table. The user can then adjust the input data based on what shows up in column 5 or accept it.
Assuming the user accepts the data, it is then used in a python script (that is already written) for much more detailed calculations.
The python script is pretty detailed and no something that is easily re-written in JScript.I can prompt the user for each data input but I would prefer to have a simple table where all the data can be entered at once
Newbie here that knows HTML, a bit of JScript and a bit of Python and some R. But not ready to take on GUI programming in TK, QT, GTK, Kivy....etc.
What is the easiest way to do this ?

You can use flask for this.
You can make a python server, and process all the inputs in python.
Here is a great tutorial to start flask
Flast tutorial

Pick a web framework (Flask seems like the simplest to recommend for your case) and implement your page as a view (whose GET request implements a view that sends the JS and web form to the browser, and whose POST request implements a view that does the detailed calculations and sends the final response).

Execute python functions and read variables from html

First off, I am very new to programming and I have a relatively basic understanding of python, and average understanding of html.
Using what I know in python, I am trying to create a basic strategy game, a bit like the likes of Age of Empires, or Command and Conquer, based on collecting resources and using it to build things, except using a simple text or button-clicking type interface. I can do a text interface fine, but its a bit boring, and I would like to use some images. I have had a 1 hour lecture on tkinter, but I have tried and failed to make anything remotely 'usable' from it. What I can do, is make decent looking html pages which would serve my purpose very well.
What I am wondering is if there is a simple way of executing python functions and calling/displaying python variables through a html page? The python functions do all the logic and present variables which represent current resource levels, production, storage capability, levels of buildings, etc. At the most basic level all I need is a way of displaying these variables, and having buttons which execute a function to say, upgrade a building, which recalculates production and all that, and returns the new set of values.
As a really simple example:
<p> Wheat production: *python integer representing production*</p>
<button type="button" onclick="*execute python variable*">Upgrade Wheat</button>
There would also, I imagine, be a need to somehow update the variables which are changed on the html page. So the button executes a function to upgrade wheat production, python now has a new value for the wheat production variable, and this needs to be updated on the page, whether this is automatic, or by some other method. I guess the simple way would be if pressing the button could also reload the page, but that seems a little clumsy.
Does anyone know of a simple way of doing this? Or perhaps a python library which might help me here?

Yes, there is a way of doing this.
There are two approaches to this. One is to have all the work done on the server, the other approach is to use Javascript.
The first approach is this: write a python script that generates your HTML. If you use Django, you will get a lot of work done for you, but you will also get a lot of stuff you don't want. Django does have a built-in template language. Django is beyond the scope of this answer. You will get to do exactly what you describe above; an example of a template might be <p> Wheat production: {{wheat_production}}</p> - your python code will set up a dict mydict={"wheat_production":10} and you will pass the name of your template file and the dict to a function which will spit out your page. You will also have to learn about HTML forms, if you haven't done so yet.
The other approach is to use Ajax - Javascript that, when your page is displayed (and, perhaps, when buttons are clicked, or at regular intervals) will send/receive some data to allow you to update your page. I suggest looking into JQuery to do some of the lifting for you. This means that you can update bits of the page without having to reload the entire thing. You will still have to write some code on the server to talk to the database, and send the output, usually as JSON, back to the client.
When writing this sort of thing, make sure all of your security is on the server side, and don't trust anything the user tells you. For example, if you store the number of gold pieces in a field on your form, it's going to take someone about 10 seconds to give themselves as much gold as they want. Similarly, if a player can sell a diamond for 20 gold pieces, make sure they have the diamond before giving them the gold pieces - you don't want to end up with a player with 1,000,000 gold pieces and negative a thousand diamonds. Javascript is 100% insecure, anything that Javascript can do, the player can also do.

Take a look at http://pyjs.org/
What is pyjs?
pyjs is a Rich Internet Application (RIA) Development Platform for
both Web and Desktop. With pyjs you can write your JavaScript-powered
web applications entirely in Python.
pyjs contains a Python-to-JavaScript compiler, an AJAX framework and a
Widget Set API. pyjs started life as a Python port of Google Web
Toolkit, the Java-to-JavaScript compiler.
You can compile Python programs to javascript, and also use their Python libraries to generate HTML. Here's an example from their getting started guide:
from pyjamas import Window
from pyjamas.ui import RootPanel, Button
def greet(sender):
Window.alert("Hello, AJAX!")
class Hello:
def onModuleLoad(self):
b = Button("Click me", greet)
RootPanel().add(b)

build a web page at runtime with python?

I am trying to build static pages, to display data on my internal website.
I would like to grab data from various text files, so every time that new data is created; I simply need to run again the script and the page is created with the new data.
I can't use JS or other runtime languages since my server allows only static pages; so I opted for python, to build the static pages.
Now the question is: how do I write such script, that allow me to build a web page?
All the data that I need is 3-4 lines, so the page is not so complex. I tried to create an empty page, and then try to modify the content via python but it was a disaster; then I thought that it would be probably simpler to build the whole page from scratch every time.
To be clear, I am making a simple page with white background, and some text on it, adjusted so it is nice to read; no graphic, no animations, nothing; just pure old school HTML.
Is there a template to do what I am trying to achieve? Thanks

You mean something like this?
I'm using the yattag library to define the template.
from yattag import Doc
def homepage_content():
return {
'text': open('/home/username/texts/homepage_text.txt').read(),
'title': open('/home/username/texts/homepage_title.txt').read()
}
def page_template(content):
doc, tag, text = Doc().tagtext()
with tag('html'):
with tag('head'):
with tag('title'):
text(content['title'])
with tag('body'):
with tag('div', id = 'main'):
text(content['text'])
return doc.getvalue()
def create_homepage():
with open('/home/username/www/index.html', "w") as fp:
fp.write(page_template(homepage_content()))

Bidirectional flow between D3.js frontend and Python Backend? / Interactive graphs in a website

So this is somewhat similar to
What's easiest way to get Python script output on the web?
and
Matplotlib: interactive plot on a web server
but none of those used d3.js and I don't think they achieve the same level of interactive-ness.
So I am a newcomer to d3.js and frankly have no clue where I should start in this instance.
Project Flow:
Ask user name on the front end. Send this to backend python
get graph data (object + edges) for a depth of x where x is distance away from a starting node by Python calls to 3rd party website API
run some machine learning on the data (python)
display the graph + some numbers (maybe in the bottom right corner) in d3.js
loop:
Have an interactive d3.js graph so that if I click a node, it will redo the calculation with that as the starting node
have it so that d3.js shows part of the text (first 10 chars?) of each node and then shows the full text once you mouse over.
(7) Bonus updating embeddable picture? Like those profile signatures?
Questions:
How would I achieve this bidirectional communication?
Where should I start?
Is there a better way I can do this?
Sidenote: I'm planning to do this project using Google App Engine, I don't know if that lets you use it as a 'vm' though.
Thanks!
Edit: This looks a lot like Python at backend JS at frontend integration, but I don't know where to even start and I think they suggest more art-oriented tools as a result of his project.

It sounds like you need to write a web service in Python that consumes and returns JSON. I'm not sure if the data is originating on the client or the server, so for this example, I'm assuming it comes from the client.
POST your graph data in JSON format to your webservice
Do processing server-side
return JSON representing the results of the processing to the client
Render it with d3.js
User interaction on the client creates a new or modified data structure
Go back to step 1
Maybe your server-side web service method looks like this (this is pseudo-code based on web.py):
import simplejson
class ml(object):
def POST(self,data=None):
# turn the JSON string into a Python data structure
d = simplejson.loads(data)
# do stuff
results = alter_data(d)
web.header('Content-Type','application/json')
# send JSON back to the client
return simplejson.dumps(results)
And your client-side code might look something like this (again, just a rough sketch). This example assumes that you're using jQuery to do your AJAX stuff.
function render(data,textStatus) {
d3
.select('svg')
.selectAll('circle')
.data(data)
.enter()
.append('circle')
.on('click',function() {
$.getJSON(
'your_webservice_url',
// POST an updated data structure to your service
{ 'data' : alter_data(data) },
// Call the render method again when new data is received
render);
});
}
d3.js' enter and exit methods make updating the visual representation of your data very easy. I recommend reading the documentation for those.
This example might give you some ideas about how to represent your graph data with JSON, and how to render it.
Check out the on method for an idea about how to trigger a POST to your web service when a node is clicked.
I'm going to skip the really specific stuff, like displaying text with d3.js. I'm sure you can figure out how to do that by perusing the d3.js documentation.

This is a late answer and I've just stumbled across this python package: Bokeh
To cover points 1-6 of your requirements it seems you pretty much only have to write python code which will be a plus for someone not used to writing java script.
But then this package is still quite new and might have some limitations I'm not yet aware of at the moment.