I'm trying to pass a variable from one page to another using google app engine, I know how to pass it using GET put putting it in the URL. But I would like to keep the URL clean, and I might need to pass a larger amount of data, so how can pass info using post.
To illustrate, I have a page with a series of links, each goes to /viewTaskGroup.html, and I want to pass the name of the group I want to view based on which link they click (so I can search and get it back and display it), but I'd rather not use GET if possible.
I didn't think any code is required, but if you need any I'm happy to provide any needed.
Links inherently generate GET requests. If you want to generate a POST request, you'd need to either:
Use a form with method="POST" and submit it, or
Use AJAX to load the new page.
Related
I'm very new to learn about web scraping. By using xpath selector i am trying to get the knowledge on that webpage : https://seffaflik.epias.com.tr/transparency/uretim/planlama/kgup.xhtml
But the point is, whenever you change the date or the powerplant name, URL does not change therefore when you fetch the response, you are getting always the same and wrong answer. Is there a way to find the correct URL or anything else related to HTML Markup etc ?
For a scraping operation like this, you'll need to do a bit more than just load the document and then grab the content. The document in-question relies on JavaScript to load new information from some other resource after the user has defined a particular set of parameters and updated the form.
After loading the document, you'll need to define your search parameters. You can do this via JavaScript injection or via your browser's console. For example, if you were trying to define the value for the first date field, you could use
document.querySelectorAll('#j_idt199 input')[1].value = "Some/New/Date";
Repeat this process for the other fields you wish to define in your search, and then run the following code to programmatically execute your search:
document.querySelector('#j_idt199 button').click();
After that, you can either grab the information you want using plain JS query selectors, or you can implement a scraping library like artoo.js to help you interpret the data and export it.
As all we know in web application we have get method and post data method.
Here my problem appear with post data.
For example i want to make my python code that access for search bar of website by insert same values and submit (the website button), then check for the page.
How the code gonna be then if there any documentation about this python concepts!
I am totally confused
Note : i am just beginner in python.
If the website relies on javascript, you're going to need to use something like Selenium which will emulate a typical browser and allow you to insert information onto a page and execute javascript commands.
If, however, the search bar simply posts data to a URL. You can determine that URL and then use requests to post the data and retrieve the result.
resp = requests.post('http://website/search', data = {'term':'value'})
In Django whenever I have to use site navigation I use the Reverse resolution of URLs. This way I can make django -render- each new html page and pass that page whatever arguments I want to through the views.
However I am wondering how should I do this in web.py. For instance, I have a web.py template that contains a variable $user. At some point in the main webpage a simple button contains a link of the form
Account
which redirects a user to his account page. Now, I need to pass $user on to account.html so that he/she can change his/her details. The problem is that since I can't directly link to account.html cause it's not a static page, how should I go through web.py and use its render method?
Thanks in advance.
It seems to me that the best way to do this would be with a session variable. That kind of variable can be consistent across several pages.
I'm a noob at this so can anyone tell me how to login to a website and fill forms and retrieve results that can be parsed into say .csv. For instance, a website where you enter certain parameters and the server returns products that best match your input parameters. I need to retrieve the list of products with their specifications and parse them into .csv. Doing that requires me to select certain buttons on the webpage which seem to be javascript objects. I tried mechanize but it doesn't seem to work for javascript objects. I would prefer to post my queries in python. Thanks!
You have two options I can think of:
1) Figure out what the fields generated by the javascript are named, or their naming scheme, and submit the values directly to the form handler -- eliminating the need to even deal with the input page.
2) Use a framework that emulates a "real" browser and is capable of processing javascript. This question has some suggestions for frameworks, but having never used one, I can't suggest any myself.
I have an app that uses htmlWindow and would like to migrate it to the new webview found in wx 2.9. However, I have learned there is no built-in method available to pass a JavaScript variable from the webpage back to the Python code. There is a RunScript method that allows one to send JavaScript to the page, but no method to retrieve an element id on a user click or any other user interaction.
My question is, is there any workaround to this? Is there any way to intercept, say, an alert call or anything else and get the data? The webview display is of not much value if one cannot receive data from user interaction.
As far as I'm aware the only way to get a return value from RunScript() is to use the page title hack.
e.g. somewhere in RunScript you set document.title to the value you wish to retrieve and get it into python with GetCurrentTitle(), if you wish you can reset the title after you have retrieved the data.
So if self.html is the webview
self.html.RunScript("""
//javascript goes here
// variable we wish to retrieve is called return_value
document.title = return_value
""")
r = self.html.GetCurrentTitle()
If you want to initiate it from within the webview it can be done (as suggested in the link Robin posted) by overriding the wxEVT_COMMAND_WEB_VIEW_NAVIGATING so that when it receives a custom url scheme, e.g. retrievedata://data/.... it retrieves the data from the url and does whatever you want with it, making sure you call evt.Veto() at some point. You can then pass any data you wish by simply calling a suitable url from within javascript.
Totally untested code below (just to give you an idea of how it can be done)
def OnPageNavigation(self, evt):
url = evt.GetUrl()
if url.startswith("retrievedata://data/"):
data = url[len("retrievedata://data/"):]
evt.Veto()
// Do whatever you want with the data
Alternatively you could use a combination of the two ideas and create a single url that when accessed calls GetPageTitle() and just make sure you set document.title before calling the page.
There was recently some discussion on the wx-users mail list and a suggestion for a workaround for things like this. I haven't tried it myself, but you may find it useful. See https://groups.google.com/d/topic/wx-users/Pdzl7AYPI4I/discussion