I am trying to create a script for myself to use on a few classified sites and starting with cl, I am using flask web framework and robobrowser but not going so well.
The Goal It will take my preset values and put them in the fields from that classifieds websites. Doesnt seem like a difficult concept however after 5 hours of reading online different code and trial and error I remembered the best developers are on stack...
I should inform you I am new to Python and still have alot to learn so any help would be greatly appreciated.
The error I get is:
assert isinstance(form, 'str')
TypeError: isinstance() arg 2 must be a type or tuple of types
but I dont see how to fix this and completely lost. HELP!!!
thanks in advance
# autosubmit to site
from flask import Flask
from robobrowser import RoboBrowser
app = Flask(__name__)
#app.route('/', methods = ['GET', 'POST'])
class My_RoboBrowser(RoboBrowser):
def __init__(self, auth=None, parser=None, headers=None, user_agent=None, history=True):
RoboBrowser.__init__(self, parser=None, user_agent=None, history=True)
def Open(self, vURL, vVerify=True):
response = self.session.get(vURL, verify=vVerify)
self._update_state(response)
browser = My_RoboBrowser(RoboBrowser, "html.parser");
urlL = 'https://accounts.craigslist.org/login'
browser.Open(urlL)
form = browser.get_form(id='login')
assert isinstance(form, 'str')
form['username'] = 'username'
form['password'] = 'password'
browser.submit_form(form)
urlQ = 'https://post.craigslist.org/k/qGDv7K4q5RGD0B5ZEBgXOQ/GLzgd?s=edit'
browser.open(urlQ)
#Question_Tag = browser.find_all(class_="not_answered")[0]
#ID = Question_Tag.get('data-qid')
#Get the form to fill out
Form = browser.get_form(id='formpost')
Form['PostingTitle'].value = 'Create this advertise ment in py'
Form['Postal_code'].value = ['10543']
Form['PostingBody'].value = 'TOGETHER WE INNOVATE Stress free communication with developers that are in the United States. We pride ourselves in bringing your web and app ideas to life and keeping your data secured'
browser.submit_form(Form)
if __name__ == "__main__":
app.run()
isinstance returns true if the first argument is an instance (or subclass) of the second argument otherwise false. In your assertion the form variable is of type robobrowser.forms.form.Form. You can see this with the following code:
print(type(form)) # should print <class 'robobrowser.forms.form.Form'>
Your particular assertion will pass if you update the second argument to indicate this robobrowser Form class:
# Add the following to your imports
import robobrowser
# ... existing code below imports
# The following should be true if form is valid
assert isinstance(form, robobrowser.forms.form.Form)
You can also import the Form class directly but you'll have to update the assertion accordingly:
from robobrowser.forms.form import Form
# ... existing code below imports
assert isinstance(form, Form)
Edit:
Below is the code to properly get the form from that page. There are no forms with an id of 'login' but you can select it by using its action or grabbing the first form on the page.
# ... previous code ...
form = browser.get_form(action='https://accounts.craigslist.org/login')
# you can also use: form = browser.get_form(0)
# It looks like you aren't selecting the fields by their proper names either.
form['inputEmailHandle'] = 'fill in username here'
form['inputPassword'] = 'fill in password here'
browser.submit_form(form)
See if the added code above helps.
Related
Im trying to use python and flask so that i can have someone submit a report (via ajax)
I got the items from my form in the python view, but now im trying to export it to my viewmodel so that from within my viewmodel i can call a function, however, im stuck as to how to do this with or continue...
This is my view
#bp.route("/rapid7/submit_report", methods=["GET"])
#login_required_with_ip_whitelist
def submit_report():
log_request()
vm = Rapid7()
return render_template("rapid7/submit_report.html", **vm.to_dict())
#bp.route("/rapid7/submit_report/process", methods=["POST"])
#login_required_with_ip_whitelist
def process():
log_request()
getFormData = (request.form)
reportName = (getFormData['reportName'])
districtName = (getFormData['districtName'])
rapid7Query = (getFormData['rapid7Query'])
rapid7SeverityLevel = (getFormData['rapid7SeverityLevel'])
print(reportName, districtName, rapid7Query, rapid7SeverityLevel)
r7report = Rapid7.check_action(reportName, districtName, rapid7Query, rapid7SeverityLevel)
print(r7report)
#exort to viewmodel
#r7_insightvm_report.configure_report(reportName, districtName, rapid7Query, rapid7SeverityLevel)
return reportName, districtName, rapid7Query, rapid7SeverityLevel
my viewmodel
from shutil import ExecError
from app.viewmodels.rapid7.rapid7 import rapid_7_report
from flask import current_app
from services.rapid7 import insightvmreport
from app.viewmodels.shared.viewmodelbase import ViewModelBase
class Rapid7(ViewModelBase):
def __init__(self):
super().__init__()
self.title = "Rapid7 Test"
self.check_action()
def check_action(self,reportName, districtName, rapid7Query, rapid7SeverityLevel):
try:
self.response = insightvmreport.check_action(
#path = current_app.config["r7_host_path"],
districtName=districtName,
reportName=reportName,
rapid7Query=rapid7Query,
rapid7SeverityLevel=rapid7SeverityLevel,
site_scope={"site":"DISTRICT 1 (EUR)","scan":"DISCOVERY"},
version="2.3.0",
action="download",
)
except Exception as e:
print(f"Failed to connect Error message: {e}")
when i run my flask app, it says
NameError: name 'reportName' is not defined
i want to use the reportName from the view (from what the user submitted to the post request), if i do a print reportName, i can see that the user was able to submit the data
i kind of understand that i have Rapid7() in my view (under the get request), which could be part of the problem, but im wondering if there's a way around this or a way to fix it, in order for my ajax script to work the way i want (Reload without having to refresh), id like for it to load the class, if this wont work is there a better solution?
I'm a new in django. I want to
bulid a webapp to search map by address.(Geocoding) This is my view.py, I want to response to my html file to show the map for the input address. My code shows it has name error at/map/. I dont know the reason. Thanks for kindly reply.
from django.shortcuts import render
import urllib
from urllib.request import urlopen
import json
def searchMap(request):
if request.method == "POST":
global latitude
global longitude
city_name = request.POST.get('address')
city_name_Url="https://maps.googleapis.com/maps/api/geocode/json?
address"+city_name
city_name_Url_Quote=urllib.parse.quote(city_name_Url,':?=/')
response=urlopen(city_name_Url_Quote).read().decode('utf-8')
response_json = json.loads(response)
latitude = response_json.get('results')[0]['geometry']['location']['lat']
longitude = api_response_dict('results')[0]['geometry']['location']['lng']
return render(request,'WebPage1.html',{'Latitude':latitude,'Longitude':longitude})
The error message:
NameError at /map/
name 'latitude' is not defined Request Method: GET
Request URL: http://127.0.0.1:8000/map/ Django Version: 1.8.13
Exception Type: NameError Exception Value: name 'latitude' is not
defined Exception
Location: C:\Users\alienware\Desktop\DjangoWebProject12\DjangoWebProject12\HelloWorld\views.py
in searchMap, line 26 Python
Executable: C:\Users\alienware\Desktop\DjangoWebProject12\DjangoWebProject12\env_DjangoWebProject2\Scripts\python.exe
Python Version: 3.6.3 Python Path:
['C:\Users\Desktop\DjangoWebProject12\DjangoWebProject12',
'C:\Users\Desktop\DjangoWebProject12\DjangoWebProject12',
'C:\Users\Desktop\DjangoWebProject12\DjangoWebProject12\env_DjangoWebProject2\Scripts\python36.zip',
'C:\Users\AppData\Local\Programs\Python\Python36\DLLs',
'C:\Users\AppData\Local\Programs\Python\Python36\lib',
'C:\Users\AppData\Local\Programs\Python\Python36',
'C:\Users\Desktop\DjangoWebProject12\DjangoWebProject12\env_DjangoWebProject2',
'C:\Users\Desktop\DjangoWebProject12\DjangoWebProject12\env_DjangoWebProject2\lib\site-packages']
Server time: Mon, 5 Feb 2018 21:57:22 +0800
I assume (from the error traceback) that your real code looks like this:
def searchMap(request):
if request.method == "POST":
# XXX totally unrelated but : __NEVER__ use mutable globals
# in a django app.
global latitude
global longitude
city_name = request.POST.get('address')
city_name_Url="https://maps.googleapis.com/maps/api/geocode/json?
address"+city_name
city_name_Url_Quote=urllib.parse.quote(city_name_Url,':?=/')
response=urlopen(city_name_Url_Quote).read().decode('utf-8')
response_json = json.loads(response)
latitude = response_json.get('results')[0]['geometry']['location']['lat']
longitude = api_response_dict('results')[0]['geometry']['location']['lng']
return render(request,'WebPage1.html',{'Latitude':latitude,'Longitude':longitude})
Now ask yourself what happens when the request's method is not POST... Yes, everything in the if block is ignored, and only the last statement (return render(...)) is executed. At this point, neither latitude nor longitude have been defined, hence your error.
The first thing to fix is the use of a POST request for a search. POST is used to update the server's state. A search does not change the server's state (at least it should not, and your's doesn't indeed) so the correct verb here is GET. As an added bonus, it will make your search result page bookmarkable.
So first change your template code to use GET as value for your form's method attribute. Then in your view, don't test on the request method at all and look for querystring args in request.GET. Also, you'll want to handle the case where the user didn't actually send anything:
def searchMap(request):
context = {}
city_name = request.GET.get('address', '').strip()
if city_name:
# hint: use the `python-requests` module instead,
# it will make you life much easier
city_name_Url="https://maps.googleapis.com/maps/api/geocode/json?
address"+city_name
city_name_Url_Quote=urllib.parse.quote(city_name_Url,':?=/')
response=urlopen(city_name_Url_Quote).read().decode('utf-8')
response_json = json.loads(response)
context["Latitude"] = response_json.get('results')[0]['geometry']['location']['lat']
context["Longitude"] = api_response_dict('results')[0]['geometry']['location']['lng']
else:
# here you want to display an error message to
# the user - don't forget to check the case
# in your template. Note that it would be simpler
# using a Django Form...
context["error"] = "Some errorr message here"
return render(request,'WebPage1.html',context)
So i been trying to get this to work but at the same time i do not understand some of these code means. I'm sorry for making the question so long but i want to understand how these works.
I am trying to make a HTTP request to another API to do POST and GET method using django. Based on the website code example, which is this url: https://www.twilio.com/blog/2014/11/build-your-own-pokedex-with-django-mms-and-pokeapi.html
As i wanted to use HTTP Request on my API to call other API, therefore i wanted to get a better understanding of how these works and how to use it.
The code is at the bottom of the website. But i will just provide the code here so it is easier for you.
website code
from django_twilio.views import twilio_view
from twilio.twiml import Response
import requests
import json
BASE_URL = 'http://pokeapi.co'
def query_pokeapi(resource_uri):
url = '{0}{1}'.format(BASE_URL, resource_uri)
response = requests.get(url)
if response.status_code == 200:
return json.loads(response.text)
return None
#twilio_view
def incoming_message(request):
twiml = Response()
body = request.POST.get('Body', '')
body = body.lower()
pokemon_url = '/api/v1/pokemon/{0}/'.format(body)
pokemon = query_pokeapi(pokemon_url)
if pokemon:
sprite_uri = pokemon['sprites'][0]['resource_uri']
description_uri = pokemon['descriptions'][0]['resource_uri']
sprite = query_pokeapi(sprite_uri)
description = query_pokeapi(description_uri)
message = '{0}, {1}'.format(pokemon['name'], description['description'])
image = '{0}{1}'.format(BASE_URL, sprite['image'])
frm = request.POST.get('From', '')
if '+44' in frm:
twiml.message('{0} {1}'.format(message, image))
return twiml
twiml.message(message).media(image)
return twiml
twiml.message("Something went wrong! Try 'Pikachu' or 'Rotom'")
return twiml
My question is:
i have read about the request.POST and request.POST.get but i still don't get it. Isn't request.POST = POST method/create function ?
what does body.lower mean ? Cant seems to find anything about it.
I am very confuse about this part
sprite_uri = pokemon['sprites'][0]['resource_uri']
description_uri = pokemon['descriptions'][0]['resource_uri']
sprite = query_pokeapi(sprite_uri)
description = query_pokeapi(description_uri)
is pokemon['sprites'] refers to the sprites field in the api ?
What does this even means ?
frm = request.POST.get('From', '')
if '+44' in frm:
twiml.message('{0} {1}'.format(message, image))
return twiml
twiml.message(message).media(image)
return twiml
request.POST.get('From', '') Isn't POST where user enter data ? Where does 'From' come from? And what does this means ? if '+44' in frm: if +44 is found in frm ?
ALL The questions are based on very basic python concepts, I recommend you go through python docs here Python Docs
Diff in request.POST and request.POST.get()
Ex request.post has following dict {'abc_key': 'abc_value'}
than request.POST['abc_key'] will give 'abc_value'
but request.POST['xyz_key'] will throw error
so we use default value to escape this error
request.POST.get('xyz_key', "default_value")
this will not give error if xyz_key is not found
body.lower
This method returns a copy of the string in which all case-based
characters have been lowercased.
check this link lower()
pokemon['sprites'][0]['resource_uri']
this is serching in pokemon (which have dictionary values)
Ex. pokemon = {'sprites':[{'resource_uri': 'res_value'}, 1, 2, 3 ]}
so pokemon['sprites'][0]['resource_uri'] will give 'res_value'
frm = request.POST.get('From', '') same as i said in 1st point
if '+44' in frm:
this will return True if string '+44' is substring in frm
variable(string)
I'm developing a webapp using Google App Engine and Python.
I'm facing a strange problem and i don't know how to solve it and what causes it.
When I fill a form I send the data for checking them. If they aren't complete and some fields are missed the server send the form back with an advice "FILL ALL THE FIELDS!".
That's work pretty well.
What I'm trying to do is sending the form back with the "description" and "title" fields filled with what the user has written before submitting the form, so he must to fill only the unfilled fields (and he doesn't need to rewrite everything from the beginning).
That's the code:
class SaleAnnCheck(BaseHandler):
def post(self):
title = self.request.POST.get('title')
cat = self.request.POST.get('cat')
description = self.request.POST.get('description')
AcqOpt = self.request.POST.get('AcqOpt')
lat = self.request.POST.get('lat')
lng = self.request.POST.get('lng')
image1 = self.request.POST.get("file1", None)
image2 = self.request.POST.get("file2", None)
image3 = self.request.POST.get("file3", None)
logging.info("info sale announcment")
logging.info(title)
logging.info(cat)
logging.info(description)
logging.info(AcqOpt)
logging.info(lat)
logging.info(lng)
logging.info(image1)
logging.info(image2)
logging.info(image3)
self.session["ECommerceUser"]["description"]=description
self.session["ECommerceUser"]["title"]=title
logging.info('session')
logging.info(self.session["ECommerceUser"])
if title == '' or cat == None or description == '' or AcqOpt == None or lat == '' or lng == '':
error = 'empty'
urlR='/CreateSaleAnn?'+urllib.urlencode({'error':'empty'})
self.redirect(urlR)
class Create(BaseHandler):
def get(self):
error = self.request.get('error')
if error == 'empty':
logging.info('sbagliato')
logging.info(self.session["ECommerceUser"])
template = JINJA_ENVIRONMENT.get_template('templates/CreateAnnouncement.html')
w = self.response.write
w(template.render({'error':'FILL ALL THE MANDATORY FIELDS!', 'description': self.session["ECommerceUser"]["description"], 'title': self.session["ECommerceUser"]["title"]}))
else:
logging.info('giusto')
logging.info(self.session["ECommerceUser"])
template = JINJA_ENVIRONMENT.get_template('templates/CreateAnnouncement.html')
w = self.response.write
w(template.render({'description': self.session["ECommerceUser"]["description"], 'title': self.session["ECommerceUser"]["title"]}))
When I submit the form the content is checked by making an HTTP post request to a certain URL, handled by SaleAnnCheck.
Description and Title are saved in the session correctly (i checked it by printing the content of self.session["ECommerceUser"] in the logs). Then, if a field isn't filled, the server redirect again to the form page, by a GET request to a related URL.
The requests to that URL is handled by Create. But when i try to render the HTML template of the form (using jinja2) with the previous typed values of Description and Title the related text areas are not filled with that values.
It happens because self.session["ECommerceUser"]["description"] and self.session["ECommerceUser"]["title"] are empty, but they weren't when i checked them before (in SaleAnnCheck).
Why it happens? Any explanation? It's a weird problem and there aren't any tips or suggestion about on internet
This is because 'self.session' is not a Session, it's just a class variable and will not be readable outside the class. If you really want to use persistent sessions for storing variables, try something like this:
From the docs:
http://docs.python-requests.org/en/master/user/advanced/
s = requests.Session()
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get('http://httpbin.org/cookies')
I'm brand new at Python and I'm trying to write an extension to an app that imports GA information and parses it into MySQL. There is a shamfully sparse amount of infomation on the topic. The Google Docs only seem to have examples in JS and Java...
...I have gotten to the point where my user can authenticate into GA using SubAuth. That code is here:
import gdata.service
import gdata.analytics
from django import http
from django import shortcuts
from django.shortcuts import render_to_response
def authorize(request):
next = 'http://localhost:8000/authconfirm'
scope = 'https://www.google.com/analytics/feeds'
secure = False # set secure=True to request secure AuthSub tokens
session = False
auth_sub_url = gdata.service.GenerateAuthSubRequestUrl(next, scope, secure=secure, session=session)
return http.HttpResponseRedirect(auth_sub_url)
So, step next is getting at the data. I have found this library: (beware, UI is offensive) http://gdata-python-client.googlecode.com/svn/trunk/pydocs/gdata.analytics.html
However, I have found it difficult to navigate. It seems like I should be gdata.analytics.AnalyticsDataEntry.getDataEntry(), but I'm not sure what it is asking me to pass it.
I would love a push in the right direction. I feel I've exhausted google looking for a working example.
Thank you!!
EDIT: I have gotten farther, but my problem still isn't solved. The below method returns data (I believe).... the error I get is: "'str' object has no attribute '_BecomeChildElement'" I believe I am returning a feed? However, I don't know how to drill into it. Is there a way for me to inspect this object?
def auth_confirm(request):
gdata_service = gdata.service.GDataService('iSample_acctSample_v1.0')
feedUri='https://www.google.com/analytics/feeds/accounts/default?max-results=50'
# request feed
feed = gdata.analytics.AnalyticsDataFeed(feedUri)
print str(feed)
Maybe this post can help out. Seems like there are not Analytics specific bindings yet, so you are working with the generic gdata.
I've been using GA for a little over a year now and since about April 2009, i have used python bindings supplied in a package called python-googleanalytics by Clint Ecker et al. So far, it works quite well.
Here's where to get it: http://github.com/clintecker/python-googleanalytics.
Install it the usual way.
To use it: First, so that you don't have to manually pass in your login credentials each time you access the API, put them in a config file like so:
[Credentials]
google_account_email = youraccount#gmail.com
google_account_password = yourpassword
Name this file '.pythongoogleanalytics' and put it in your home directory.
And from an interactive prompt type:
from googleanalytics import Connection
import datetime
connection = Connection() # pass in id & pw as strings **if** not in config file
account = connection.get_account(<*your GA profile ID goes here*>)
start_date = datetime.date(2009, 12, 01)
end_data = datetime.date(2009, 12, 13)
# account object does the work, specify what data you want w/
# 'metrics' & 'dimensions'; see 'USAGE.md' file for examples
account.get_data(start_date=start_date, end_date=end_date, metrics=['visits'])
The 'get_account' method will return a python list (in above instance, bound to the variable 'account'), which contains your data.
You need 3 files within the app. client_secrets.json, analytics.dat and google_auth.py.
Create a module Query.py within the app:
class Query(object):
def __init__(self, startdate, enddate, filter, metrics):
self.startdate = startdate.strftime('%Y-%m-%d')
self.enddate = enddate.strftime('%Y-%m-%d')
self.filter = "ga:medium=" + filter
self.metrics = metrics
Example models.py: #has the following function
import google_auth
service = googleauth.initialize_service()
def total_visit(self):
object = AnalyticsData.objects.get(utm_source=self.utm_source)
trial = Query(object.date.startdate, object.date.enddate, object.utm_source, ga:sessions")
result = service.data().ga().get(ids = 'ga:<your-profile-id>', start_date = trial.startdate, end_date = trial.enddate, filters= trial.filter, metrics = trial.metrics).execute()
total_visit = result.get('rows')
<yr save command, ColumnName.object.create(data=total_visit) goes here>