I am looking to create a very basic website that has a single feature. This feature works as follows:
1) The user inputs a URL to a website
2) My website will scrape the given website for necessary information
3) Apply whatever logic I need and return the processed information as a string
4) Allow user to download returned data in .txt format.
I am working on this project with one collaborator, and he has created the scraping and logic (steps 2 and 3) in Python. I have looked around stack overflow to see if it is possible to execute python scripts in a website, and the consensus seemed to be that I will require a server that executes python, and then make HTTP requests to my server.
Unfortunately, I'm quite a junior developer and lack a lot of understanding regarding web dev, and my attempts to "dive in" have left me with more questions than answers. We have a deadline that is approximately 3 days from now, and the only part that remains is to set up a server that can execute a specific python script upon HTTP requests.
I think that the source of my confusion is a lack of understanding regarding how exactly a server works, and was hoping that the kind folks here at stack overflow could help me in:
1) better understanding what's going on "under the hood" on the server side after an HTTP request arrives (or better yet, how does an HTTP request even arrive at all?)
2) Explain to me like I'm 5 what I'll need to do to deploy my website - namely all the in-between steps that receive little attention from the other posts here at stack overflow.
Some example questions that run through my head are below:
What type of server will I need?
How will I know if it can run Python?
How does the server know that I want to execute a certain script, while upon a user entering a my website's homepage URL, land at the home screen?
I've also read up on "middle layers" such as 'CGI' - what does this accomplish?
Is setting up a server reinventing the wheel?
enter code here
Is there already a service out there that does what I'm looking for?
I've accomplished a similar project overnight with a Python web framework called Django, simply by following their official tutorial: https://docs.djangoproject.com/en/1.7/intro/tutorial01/
Django abstracts away all the web stuff for you and there's plenty of documentation on deploying it in various environments. With such deadlines I suggest you to just follow the tutorial above and eventually adapt the example to your use case testing it on Django's built-in web server, then follow a guide on deploying Django projects in your production environment.
Related
Im creating a server in Python that receives POST requests, process the information in the request using some scripts (sometimes using a database) and send back a answer in JSON format. Im searching for a way to run this server and code in the cloud, in a way that i dont need my PC turned on for it to work, because my connection is very unstable.
There are a lot of web hosting companies out there, you just need to find the one that is right for you.
My personal favorite for python apps is heroku, but there are many out there. AWS is another popular one.
In future when asking questions, try to do more research before hand, and try to be more specific with questions. It would have been useful to know what kind of database you are using, or whether you're using flask or django.
Is there a way to upload questions to "Retrieve and Rank" (R&R) using cURL and have them be visible in the web tool?
I started testing R&R using web tool (which I find very intuitive). Now, I have started testing the command line interface (CLI) for more efficient uploading of question-and-answer pairs using train.py. However, I would still like to have the questions visible in web tool so that other people can enter the collection and perform training there as well. Is it possible in the present status of R&R?
Sorry, no - there isn't a public supported API for submitting questions for use in the tool.
(That wouldn't stop you looking to see how the web tool does it and copying that, but I wouldn't encourage that as the auth step alone would make that fairly messy).
My question may look silly but I am asking this after too much search on Google, yet not have any clue.
I am using iCloud web services. For that I have converted this Python code to PHP. https://github.com/picklepete/pyicloud
Up to this, everything is working good. When authenticate using icloud username,password I am getting a list of web service URLs as part of response. Now for example to use Contacts web service, I need to use Contact web service URL and add a part to that URL to fetch contacts.
https://p45-contactsws.icloud.com:443/co/startup with some parameters.
The webservice URL https://p45-contactsws.icloud.com:443 is coming in response while authenticating. But the later part, 'co/startup' is there in the python code. I don't know how they found that part. So for some services which is there in Python code, they are working good. But I want to use few other service like https://p45-settingsws.icloud.com:443, https://p45-keyvalueservice.icloud.com:443 etc. and when I try to send request with correct parameters to this other services, I am getting errors like 404 not found or unauthorized access. So I believe that some URL part must be added to this just like contacts. If someone knows how or where can I get correct URL part, I will be really thankful.
Thanks to all in advance for their time reading/answering my question.
I am afraid there doesn't seem to be an official source for these API endpoints, since they seem to be discovered through sniffing the network calls rather than a proper guide from Apple. For example, this presentation, which comes from a forensic tools company, is from 2013 and covers some of the relevant endpoints. Note that iOS was still at versions 5 & 6 then (vs. the current v9.3).
All other code samples on the net basically are using the same set of API endpoints that were originally observed in 2012-2013. (Here's a snippet from another python module with additional URLs you may use.) However, all of them pretty much point to each other as the source.
If you'd like to pursue a different path, Apple now promotes the CloudKit and CloudKit JS solutions for registered apps working with iCloud data.
I'm looking for a bit of web development advice. I'm fairly new to the area but I'm sure there are some gurus out there willing to part with some wisdom.
Objective: I'm interested in controlling a Python application on my computer from my personal web hosted site. I know, this question has been asked several times before but in each case the requirements were a bit different from my own. To reduce the length of this post I'll summarize my objective in a few bullet points:
Personal site is hosted by a web hosting company
Site uses HTML, PHP, MySQL, Python and JavaScript, the majority of everything is coded by me from the ground up
An application that is coded in Python will run on a PC within my home and will communicate with an Arduino board
The app will receive commands from the internet to control actuation via the Arduino, and will transmit sensor data back to the site (such as temperature)
Looking for the communication to be bi-directional, fast and secure
Securing the connection between site and Python app would be most ideal
I'm not looking to connect to the Python application directly, the web server must serve as the 'middle man'
So far I've considered HTTP Post and HTML forms, using sockets (Python app would run as a web server), an IRC bot and reading/writing to a text file stored on the web server.
I was also hoping to have a way to communicate with the Python app without needing to refresh the webpage, perhaps using AJAX or JavaScipt? Maybe with Flash?
Is there something I'm not considering? I feel like I'm missing something. Thanks in advance for the advice!
Just thinking out loud for how I would start out with this. First, regarding the website itself, you can just use what's easiest to you, or to the environment you're in. For example, a basic PHP page will do just fine, but if you can get a site running in Python as well, I'd prefer using the same language all over.
That said, I'm not sure why you would need to use a hosted website? Given that you're already forced to have a externally accessible PC at home for the communication, why not run a webserver on that directly (Apache, Nginx, or even something like CherryPy should do)? That webserver can then communicate with the python process that is running to control your Arduino (by using e.g. Python's xmlrpclib). If you would run things via the hosting company, you would still need some process that can handle external requests securely... something a webserver is quite good at. Just running it yourself gives you all the freedom you want, and simplifies things by lessening the number of components in your solution.
The updates on your site I'd keep quite basic: commands you want to run can be handled in the request handlers of the webserver by just calling the relevant (xmlrpclib) calls. Dynamically updating the page is best done by some AJAX calls I reckon. Based on your story, these updates are easily put in a JSON object, suitable for periodically updating only the relevant segments of your page.
I love django, and I like flex. Django for it's cool debugging system (those yellow pages helps a lot to find bugs in my code), and flex for it possibilities.
Recently I come across a problem. If I create a form in flex and then communicate with the django server, I can't see any debugging info (when the exception happens in django).
Not sure, if there is a way to get the debugging info, because it is not accessible in command line (no error output), or in firebug....
Also I tried to create a quick html form, and post same data as I send from flex form, but it's a bit of pain to be honest.
Will be happy to listen how do you solve the problem
I've used firebug to debug the flex side of things. But I've been using json or XML for communication between the two. Since flash uses the browser to do the network stuff, the request should be visible in the net tab of firebug.
To debug the django side of things, you have a few options.
If you're using the django dev server, you can add print statements to find out what's going on.
You can write a unit test to see if the django side of things is doing what you expect it to, given known data.
You can use the pyDev debugger to run the django dev server and step through your code.
I use a combination of these to debug my code.