Good day folks
Recently, I made a python based web crawler machine that scrapes_ some news ariticles and django web page that collects search title and url from users.
But I do not know how to connect the python based crawler machine and django web page together, so I am looking for the any good resources that I can reference.
If anyone knows the resource that I can reference,
Could you guys share those?
Thanks
There are numerous ways you could do this.
You could directly integrate them together. Both use Python, so the scraper would just be written as part of Django.
You could have the scraper feed the data to a database and have Django read from that database.
You could build an API from the scraper to your Django implementation.
There are quite a few options for you depending on what you need.
Related
I am slightly new to python coding and I have a project coming up to which I've decided to make some code that when entering a Facebook users URL it will return all data that their profile has to offer. Any help would be greatly appreciated or if you have code that does similar I would love to observe.
I am looking for this to be executed in python.
I would recommend using a web scraping framework with python. There are tons of them. Beautiful Soup, Scrapy are great options. However, most web applications do have security in place to prevent you from scraping data on their platforms. I would recommend you do more research.
I have a mini project that I need to create website site where I can submit some information to a database but want to do it through Web services.
Is there any projects that I can reference or tutorial that shows me how to create a Web front end, python web service or django web services tied MS Sql backend.
I can do it in another language but really want to do it purely in python/Django or whatever framework that make sense to do it in so I can learn.
You can start off by reading the Django tutorial on official website https://www.djangoproject.com
Or read Django by examples. It will set you started with cool features and all you need to build an amazing app quickly. You will be excited after finishing the first two chapters.
For MySQL follow the documentation and it is pretty easy
I am attempting to build a web crawler to sign into FaceBook and check the online status of some family members for a project I'm building for my parents. Upon searching, I found that this is attainable through FQL queries on friend online presence, but it seems that this will be removed around April of this year. So I thought that maybe I can just do a basic crawler myself in python that will get the HTML info from online friends in my chat, but when trying to print out the HTML code after attempting to log in, it returns a very large amount of jumbled HTML and javascript that mentions "BigPipe." I see that BigPipe breaks pages into pagelets but I'm a little confused on what to make of this information.
So my questions are, does anyone know of another way to get online statuses other than the FQL queries, has anyone else attempted to crawl Facebook, has anyone attempted to crawl any site with this BigPipe response?
Thank you in advance,
Jake
You may be able to write a FireFox extension. You will not be able to scrape FB without JavaScript. That pretty much rules out most traditional scraping methods.
Using PyQt4.QtWebKit will help to deal with javascript.
Here some basic usage of it : webkit-pyqt-rendering-web-pages
Documentation: PyQt4-qtwebkit.html
I just finished my school project which requires user data from Facebook group members. I used a web crawling tool - Octoparse for data extraction, it's a non-programming application and can be used to crawl different types of data on Facebook. You can go to this tutorial:Facebook Scraping Case Study | Scraping Facebook Groups
For fun, I've learnt since last night how to do basic web scraping, using Python's urllib, urllib2, cookie-jar, and BeautifulSoup. It only took a bit, but I've figured out how to get all information from each user's profile that I need (OKCupid to be exact). However, I've only figured out how to do so, and have no idea how to go through a public database of users without an API from the site.
Is there any easy way to do so? Thanks.
I need to build a web site with the following features:
1) user forum where we expect light daily traffic
2) database backend for users to create profiles, where they can log in
and upload media (pictures)
3) users can uses their profile to buy content from an online inventory
4) create web pages, shopping carts etc for online inventory
5) secure online credit card processing
I am very familiar with python but not with python web frameworks. I do know
some SQL. How do I get started developing something like this? Is Django
a good alternative?
Not programming related per se: Where do you recommend I get web hosting with a domain
name for an application like this?
Django was made for this kind of thing. Check it out.
As far as hosting, djangofriendly.com is a great resource. I have used WebFaction before and I am absolutely in love with how easy it is to get Django going with them and with their excellent customer service. Very top notch for reasonable prices if you are going the shared hosting route.
If you are looking to speed up some of the tasks described, you should check out Pinax and Django Pluggables. Thanks to the way Django applications are setup it is trivially easy to plug an application into your project.
You can try Pylons lightweight web framework.
Your requirements make pinax sound like a library you might want to look into if you go the django route.
Google App Engine will provide hosting for free as well as Django and a db..