Develop Reference

Parametrizing a static webdriver fixture and a generator in Pytest?

I'm building an automation test for finding any possible dead links in a WP plugin. To this end, I have two helper functions.
The first spins up a Selenium webdriver:
#pytest.fixture(scope='session')
def setup():
d = webdriver.Firefox()
site = os.getenv('TestSite')
d.get(site)
login = d.find_element_by_id('user_login')
d.implicitly_wait(5)
user, pw = os.getenv('TestUser'), os.getenv('TestPw')
d.find_element(By.ID, 'user_login').send_keys(user)
d.find_element(By.ID, 'user_pass').send_keys(pw)
d.find_element(By.ID, 'wp-submit').click()
yield d, site
d.quit()
The second reads in a JSON file, picking out each page in the file then yielding it to the third function:
def page_generator() -> Iterator[Dict[str, Any]]:
try:
json_path = '../linkList.json'
doc = open(json_path)
body = json.loads(doc.read())
doc.close()
except FileNotFoundError:
print(f'Did not find a file at {json_path}')
exit(1)
for page in body['pages']:
yield page
The third function is the meat and potatoes of the test, running through the page and looking for each link. My goal is to parametrize each page, with my function header presently looking like...
#pytest.mark.parametrize('spin_up, page', [(setup, p) for p in page_generator()])
def test_links(spin_up, page):
driver, site = spin_up
# Do all the things
Unfortunately, running this results in TypeError: cannot unpack non-iterable function object. Is the only option to stick yield d, site inside some sort of loop to turn the function into an iterable, or is there a way to tell test_links to iteratively pull the same setup function as its spin_up value?

There is an easy way to do what you want, and that's to specify the setup fixture directly as an argument to test_links. Note that pytest is smart enough to figure out that the setup argument refers to a fixture while the page argument refers to a parametrization:
#pytest.mark.parametrize('page', page_generator())
def test_links(setup, page):
driver, site = setup
You might also take a look at parametrize_from_file. This is a package I wrote to help with loading test cases from JSON/YAML/TOML files. It basically does the same thing as your page_generator() function, but more succinctly (and more generally).

How to write unit testing script using unittest

I want to write a unit test cases using unittest where my input is set of images and output is set of images with boundary box in the text, also it generate co-ordinate of the boundary boxes. How can I write test script for that using unittest pyunit framework.
So I have written unittest script to test api is working or not, response should be json / and list format and response of image file format should be png,jpg format, which is working now. I have below test case scenario need to test but how I will check? I don't know.
If optional keys are not passed to API, it should not throw an error. (opposite for compulsory ones) 8) The implemented route should throw/return error if GET request is passed instead of POST. 9) If valid path but invalid file name is provided, you should see respective error. 10) In case of invalid path, you should see respective error message. 11) There is specified set of keys which are compulsory to be passed to APIs, if not it returns an error 12) verify for session time out.
Here is my code:
import requests
import json
import sshtunnel
import unittest
class TestSequentialExecutions(unittest.TestCase):
def setUp(self) -> None:
a=10
def test_API(self):
self.resp_list = []
# API url
url = ['dummy url','dummyurl']
# Additional headers.
headers = {'Content-Type': 'application/json'}
# Body
payload = [{'input': 'dummy path'},
{"path": "dummy"}]
# Test case-1 checking valid API is routed or not
# convert dict to json by json.dumps() for body data.
for i in range(len(url)):
resp = requests.post(url[i], headers=headers, data=json.dumps(payload[i], indent=4))
self.assertEqual(resp.status_code, 200)
self.resp_list.append(resp.json())
#Test case-2 to check input file is in JPG ,PNG format or not
def test_fileformat(self):
n = len(self.resp_list[1])
my_list = [1]*n
empty_list=[]
extensions = ['png','v']
for filename in self.resp_list[0]:
if filename.lower().endswith(('.png', '.jpg')):
empty_list.append(1)
else:
empty_list.append(0)
self.assertEqual(my,empy_list)
if __name__ == '__main__':
unittest.main()
Actually I am trying to write test script for below github code: https://github.com/eragonruan/text-detection-ctpn

send data to html dashboard in python flask while having dictionary in a function parameter

I want to send data to html in flask framework, i have a function which receives dictionary as a parameter, then there are several functions applied on dictionary. after that final result i want to render in to html page. Its been 48 hours i am trying from different blogs but didn't get precise solution.
imports ...
from other file import other_functions
from other file import other_functions_2
from other file import other_functions_3
app = Flask(__name__, template_folder='templates/')
#app.route("/dashboard")
def calculate_full_eva_web(input:dict):
calculate_gap = other_functions(input)
calculate_matrix = other_functions_2(input)
average = other_functions_3(input)
data = dict{'calculate_gap':calculate_gap, 'calculate_matrix':calculate_matrix,'average':average}
return render_template('pages/dashboard.html', data = data)
if __name__ == "__main__":
app.run(debug=True)
TRACEBACK

Methods that Flask can route to don't take dictionaries as inputs, and the arguments they do take need to be matches by a pattern in the route. (See https://flask.palletsprojects.com/en/1.1.x/api/#url-route-registrations)
You'd get the same error if you changed
#app.route("/dashboard")
def calculate_full_eva_web(input:dict):
to
#app.route("/dashboard")
def calculate_full_eva_web(input):
Your path forward depends on how you want to pass data when you make the request. You can pass key/value pairs via URL parameters and retrieve them via the request.args object. That might be close enough to what you want. (You'll need to remove the argument declaration from calculate_full_eva_web())
Something like
from flask import request
#app.route('/dashboard')
def calculate_full_eva_web():
input = request.args
...

unable to save data in datastore but no errors

I'm building a web crawler. some of the the data I input into datastore get saved, others do not get saved and I have no idea what is the problem.
here is my crawler class
class Crawler(object):
def get_page(self, url):
try:
req = urllib2.Request(url, headers={'User-Agent': "Magic Browser"}) # yessss!!! with the header, I am able to download pages
#response = urlfetch.fetch(url, method='GET')
#return response.content
#except urlfetch.InvalidURLError as iu:
# return iu.message
response = urllib2.urlopen(req)
return response.read()
except urllib2.HTTPError as e:
return e.reason
def get_all_links(self, page):
return re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+',page)
def union(self, lyst1, lyst2):
try:
for elmt in lyst2:
if elmt not in lyst1:
lyst1.append(elmt)
return lyst1
except e:
return e.reason
#function that crawls the web for links starting from the seed
#returns a dictionary of index and graph
def crawl_web(self, seed="http://tonaton.com/"):
query = Listings.query() #create a listings object from storage
if query.get():
objListing = query.get()
else:
objListing = Listings()
objListing.toCrawl = [seed]
objListing.Crawled = []
start_time = datetime.datetime.now()
while datetime.datetime.now()-start_time < datetime.timedelta(0,5):#tocrawl (to crawl can take forever)
try:
#while True:
page = objListing.toCrawl.pop()
if page not in objListing.Crawled:
content = self.get_page(page)
add_page_to_index(page, content)
outlinks = self.get_all_links(content)
graph = Graph() #create a graph object with the url
graph.url = page
graph.links = outlinks #save all outlinks as the value part of the graph url
graph.put()
self.union(objListing.toCrawl, outlinks)
objListing.Crawled.append(page)
except:
return False
objListing.put() #save to database
return True #return true if it works
the classes that define the various ndb Models are in this python module:
import os
import urllib
from google.appengine.ext import ndb
import webapp2
class Listings(ndb.Model):
toCrawl = ndb.StringProperty(repeated=True)
Crawled = ndb.StringProperty(repeated=True)
#let's see how this works
class Index(ndb.Model):
keyword = ndb.StringProperty() # keyword part of the index
url = ndb.StringProperty(repeated=True) # value part of the index
#class Links(ndb.Model):
# links = ndb.JsonProperty(indexed=True)
class Graph(ndb.Model):
url = ndb.StringProperty()
links = ndb.StringProperty(repeated=True)
it used to work fine when I had JsonProperty in place of StringProperty(repeated=true). but JsonProperty is limited to 1500 bytes so I had an error once.
now, when I run the crawl_web member function, it actually crawls but when I check datastore it's only the Index entity that is created. No Graph, no Listing. please help. thanks.

Putting your code together, adding the missing imports, and logging the exception, eventually shows the first killer problem:
Exception Indexed value links must be at most 500 characters
and indeed, adding a logging of outlinks, one easily eyeballs that several of them are far longer than 500 characters -- therefore they can't be items in an indexed property, such as a StringProperty. Changing each repeated StringProperty to a repeated TextProperty (so it does not get indexed and thus has no 500-characters-per-item limitation), the code runs for a while (making a few instances of Graph) but eventually dies with:
An error occured while connecting to the server: Unable to fetch URL: https://sb':'http://b')+'.scorecardresearch.com/beacon.js';document.getElementsByTagName('head')[0].appendChild(s); Error: [Errno 8] nodename nor servname provided, or not known
and indeed, it's pretty obvious tht the alleged "link" is actually a bunch of Javascript and as such cannot be fetched.
So, essentially, the core bug in your code is not at all related to app engine, but rather, the issue is that your regular expression:
'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+'
does not properly extract outgoing links given a web page containing Javascript as well as HTML.
There are many issues with your code, but to this point they're just slowing it down or making it harder to understand, not killing it -- what's killing it is using that regular expression pattern to try and extract links from the page.
Check out retrieve links from web page using python and BeautifulSoup -- most answers suggest, for the purpose of extracting links from a page, using BeautifulSoup, which may perhaps be a problem in app engine, but one shows how to do it with just Python and REs.

weird issue using twitter api to search for tweets

I've setup a code in python to search for tweets using the oauth2 and urllib2 libraries only. (I'm not using any particular twitter library)
I'm able to search for tweets based on keywords. However, I'm getting zero number of tweets when I search for this particular keyword - "Jurgen%20Mayer-Hermann". (this is challenge because my ultimate goal is to search for this keyword only.
On the other hand when I search for the same thing online (twitter interface, I'm getting enough tweets). - https://twitter.com/search?q=Jurgen%20Mayer-Hermann&src=typd
Can someone please see if we can identify the issue?
The code is as follows:
def getfeed(mystr, tweetcount):
url = "https://api.twitter.com/1.1/search/tweets.json?q=" + mystr + "&count=" + tweetcount
parameters = []
response = twitterreq(url, "GET", parameters)
res = json.load(response)
return res
search_str = "Jurgen Mayer-Hermann"
search_str = '%22'+search_str+'%22'
search = search_str.replace(" ","%20")
search = search.replace("#","%23")
tweetcount = str(50)
res = getfeed(search, tweetcount)
When I print the constructed url, I get
https://api.twitter.com/1.1/search/tweets.json?q=%22Jurgen%20Mayer-Hermann%22&count=50

I have actually never worked with the Twitter API, but it looks like the count parameter only applies to searches on timelines as a way to limit the amount of tweets per page of results. In other words, you use it with the GET statuses/home_timeline, GET statuses/mentions, and GET statuses/user_timeline endpoints.
Try without count and see what happens.

Please use urllib.urlencode to encode your query parameters, like so:
import urllib
query = urllib.urlencode({'q': '"Jurgen Mayer-Hermann"', count: 50})
This produces 'q=%22Jurgen+Mayer-Hermann%22&count=50'. Which might bring you more luck...