I am new to Python and trying to my script to send the output of this command 'ibeacon_scan -b' to be sent to a web server by query string or any other efficient way to send data continuously. Here is what the output looks like for 'ibeacon_scan -b'
iBeacon Scan ...
3F234454-CFD-4A0FF-ADF2-F4911BA9FFA6 1 4 -71 -69
3F234454-CFD-4A0FF-ADF2-F4911BA9FFA6 6 2 -71 -63
3F234454-CFD-4A0FF-ADF2-F4911BA9FFA6 1 4 -71 -69
3F234454-CFD-4A0FF-ADF2-F4911BA9FFA6 5 7 -71 -64
...keeps updating
I am piping the command to my Python script
ibeacon scan -b | stdin.py
Here is my code for the my script 'stdin.py'
#!/usr/bin/python
import fileinput
import httplib
import urllib
for line in fileinput.input():
urllib.urlencode({"UUID": {"Major":{"Minor":RSSI}}})
headers = {"Content-type": "application/x-www-formurlencoded","Accept": "text/plain"}
conn = httplib.HTTPConnection("67.205.14.22")
conn.request("POST", "post.php", params, headers)
response = conn.getrespone()
print response.status, respone.reason
data = respone.read()
print data
conn.close()
I'm getting these errors.
Traceback (most recent call last):
File "./stdin.py", line 7, in <module>
for line in fileinput.input():
File "/usr/lib/python2.7/fileinput.py", line 253, in next
line = self.readline()
File "/usr/lib/python2.7/fileinput.py", line 346, in readline
self._buffer = self._file.readlines(self._bufsize)
KeyboardInterrupt
Is my script even getting the data correctly from the pipe? Is the formatting correct for the query string?
As #TheSoundDefense pointed, it must be some KeyboardInterrupt character in the ibeacon's output:
A fast check shows that piping in linux actually works:
>>> cat tmp.txt | python -c "import fileinput; print [line for line in fileinput.input()]"
['a\n', 'b\n', 'c\n', 'd\n']
Where tmp.txt contains 4 lines with a, b, c and d.
Do you need to have it pipe in? Because if not, you can do something like this all in your python script (thanks oliver13 for the idea):
popen = subprocess.Popen(["ibeacon scan -b"], stdout=subprocess.PIPE)
for line in iter(popen.stdout.readline, ""):
urllib.urlencode({"UUID": {"Major":{"Minor":RSSI}}})
headers = {"Content-type": "application/x-www-formurlencoded","Accept": "text/plain"}
conn = httplib.HTTPConnection("67.205.14.22")
conn.request("POST", "post.php", params, headers)
response = conn.getrespone()
print response.status, respone.reason
data = respone.read()
print data
conn.close()
Related
I have the following in a script:
import requests
from json import loads
s = requests.Session()
r = s.get("https://iforgot.apple.com/password/verify/appleid", headers=headers)
headers['Sstt'] = loads([line for line in r.text.split("\n") if "sstt" in line][0])["sstt"]
headers['Content-Type'] = "application/json"
data = f'{{"id":"{email}"}}'
r = s.post("https://iforgot.apple.com/password/verify/appleid", data=data, headers=headers, allow_redirects=False).headers['Location']
headers['Accept'] = 'application/json, text/javascript, */*; q=0.01'
r = s.get(f"https://iforgot.apple.com{r}", headers=headers, allow_redirects=False).json()['trustedPhones'][0]
c = r['countryCode']
n = r['numberWithDialCode']
Whenever I run this, I receive this error:
File "/home/user/xsint/modules/apple.py", line 10, in apple
headers['Sstt'] = loads([line for line in r.text.split("\n") if "sstt" in line][0])["sstt"]
File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 10 (char 9)
But the thing that I really can't figure out is if I run each of these commands in a Python 3 console they work. Does anyone see what the problem is?
[EDITED]
You have more than one data in your json, and json.loads() is not able to decode more than one.
check below line
headers['Sstt'] = loads([line for line in r.text.split("\n") if "sstt" in line][0])["sstt"]
it must be not a json format change to something like
{
"key" :[value, value]
}
and it should work,
Python json.loads shows ValueError: Extra data
First, I think line 10 and 12 is incomplete?
LINE 10 : data, headers=headers, all$
LINE 12 : json()['trus$
Second, it will be more helpful with more error message.
I am scraping a very large website with Beautiful Soup for a project and want to use the Pool module to speed it up. I am getting a weird error where it is not correctly reading the list of URL's, as far as I can tell it is just grabbing the first 'h'.
The entire code works perfectly if I do not use pool. The list of URL's is read properly. I am not sure if there is something weird about how you have to prepare the URL's when calling p.map(scrapeClauses, links) because if I simply call scrapeClauses(links) everything works.
Here is my main function:
if __name__ == '__main__':
links = list()
og = 'https://www.lawinsider.com'
halflink = '/clause/limitation-of-liability'
link = og + halflink
links.append(link)
i = 0
while i < 50:
try:
nextLink = generateNextLink(link)
links.append(nextLink)
link = nextLink
i += 1
except:
print('Only ', i, 'links found')
i = 50
start_time = time.time()
print(links[0])
p = Pool(5)
p.map(scrapeClauses, links)
p.terminate()
p.join()
#scrapeClauses(links)
and here is scrapeClauses():
def scrapeClauses(links):
#header to avoid site detecting scraper
headers = requests.utils.default_headers()
headers.update({
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
})
#list of clauses
allText = []
number = 0
for line in links:
page_link = line
print(page_link)
page_response = requests.get(page_link, headers=headers)
html_soup = BeautifulSoup(page_response.content, "html.parser")
assignments = html_soup.find_all('div', class_ ='snippet-content')
for i in range(len(assignments)):
assignments[i] = assignments[i].get_text()
#option to remove te assignment that precedes each clause
#assignments[i] = assignments[i].replace('Assignment.','',1)
allText.append(assignments[i])
#change the index of the name of the word doc
name = 'limitationOfLiability' + str(number) + '.docx'
#some clauses have special characters tat produce an error
try:
document = Document()
stuff = assignments[i]
document.add_paragraph(stuff)
document.save(name)
number += 1
except:
continue
I did not include generateNextLink() to save space and because I am pretty sure the error is not in there but if someone thinks it is I will provide it.
As you can see I 'print(page_link) in scrapeClauses. If I am not using pool, it will print all the normal links. But if I use pool, a bunch of h's print out line after line. I then get and error that h is not a valid URL. I will show the error code below.
https://www.lawinsider.com/clause/limitation-of-liability
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
h
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\wquinn\AppData\Local\Programs\Python\Python37-32\lib\multiproce
ssing\pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "C:\Users\wquinn\AppData\Local\Programs\Python\Python37-32\lib\multiproce
ssing\pool.py", line 44, in mapstar
return list(map(*args))
File "C:\Users\wquinn\Web Scraping\assignmentBSScraper.py", line 20, in scrape
Clauses
page_response = requests.get(page_link, headers=headers)
File "C:\Users\wquinn\AppData\Local\Programs\Python\Python37-32\lib\site-packa
ges\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\wquinn\AppData\Local\Programs\Python\Python37-32\lib\site-packa
ges\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\wquinn\AppData\Local\Programs\Python\Python37-32\lib\site-packa
ges\requests\sessions.py", line 519, in request
prep = self.prepare_request(req)
File "C:\Users\wquinn\AppData\Local\Programs\Python\Python37-32\lib\site-packa
ges\requests\sessions.py", line 462, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "C:\Users\wquinn\AppData\Local\Programs\Python\Python37-32\lib\site-packa
ges\requests\models.py", line 313, in prepare
self.prepare_url(url, params)
File "C:\Users\wquinn\AppData\Local\Programs\Python\Python37-32\lib\site-packa
ges\requests\models.py", line 387, in prepare_url
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'h': No schema supplied. Perhaps
you meant http://h?
The second argument of p.map get an list. Each such element will be sent to a function. So you function got a string and not a list of string as you expect.
The minimal example is:
from multiprocessing import Pool
def f(str_list):
for x in str_list:
print ('hello {}'.format(x))
if __name__ == '__main__':
str_list = ['111', '2', '33']
p = Pool(5)
p.map(f, str_list)
p.terminate()
p.join()
Output is:
hello 1
hello 1
hello 1
hello 2
hello 3
hello 3
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 8 years ago.
Improve this question
I am getting data in this format every second or so from a bash command 'ibeacon_scan"
ibeacon scan -b | stdin.py
Output:
ibeacon scan...
3F234454-CFD-4A0FF-ADF2-F4911BA9FFA6 1 4 -71 -69
3F234454-CFD-4A0FF-ADF2-F4911BA9FFA6 6 2 -71 -63
3F234454-CFD-4A0FF-ADF2-F4911BA9FFA6 1 4 -71 -69
3F234454-CFD-4A0FF-ADF2-F4911BA9FFA6 5 7 -71 -64
I need to send that information by query string. Here is my code.
#!/usr/bin/python
import fileinput
import httplib
import urllib
for line in fileinput.input():
string = line
string2 = string.split(" ")
parmas = string2
parmas = urllib.urlencode({"UUID": "Major","Minor":"RSSI"})
headers = {"Content-type": "application/x-www-form-urlencoded","Accept": "text/plain"}
conn = httplib.HTTPConnection("67.205.14.22")
conn.request("POST", "post.php", params, headers)
response = conn.getresponse()
print response.status, response.reason
data = response.read()
print data
conn.close()
I'm getting this error:
Traceback (most recent call last):
File "./stdin.py", line 14, in <module>
conn.request("POST", "post.php", params, headers)
NameError: name 'params' is not defined
Something is wrong with params? How do I format this to correctly accept the 'ibeacon scan' command and send it by HTTP post?
for line in fileinput.input():
string = line
string2 = string.split(" ")
parmas = string2 # you call the variable parmas here
parmas = urllib.urlencode({"UUID": "Major","Minor":"RSSI"})
headers = {"Content-type": "application/x-www-form-urlencoded","Accept": "text/plain"}
conn = httplib.HTTPConnection("67.205.14.22")
conn.request("POST", "post.php", params, headers) # and params here
You have made what looks to be a typo.
In a file (say parser.py) I have:
import argparse
def parse_cmdline(cmdline=None):
parser = argparse.ArgumentParser()
parser.add_argument('--first-param',help="Does foo.")
parser.add_argument('--second-param',help="Does bar.")
if cmdline is not None:
args = parser.parse_args(cmdline)
else:
args = parser.parse_args()
return vars(args)
if __name__=='__main__':
print parse_cmdline()
Sure enough, when called from the command line it works and give me pretty much what I expect:
$ ./parser.py --first-param 123 --second-param 456
{'first_param': '123', 'second_param': '456'}
But then I want to unittest it, thus I write a test_parser.py file:
import unittest
from parser import parse_cmdline
class TestParser(unittest.TestCase):
def test_parse_cmdline(self):
parsed = parse_cmdline("--first-param 123 --second-param 456")
self.assertEqual(parsed['first_param'],'123')
self.assertEqual(parsed['second_param'],'456')
if __name__ == '__main__':
unittest.main()
Then I get the following error:
usage: test_parser.py [-h] [--first-param FIRST_PARAM]
[--second-param SECOND_PARAM]
test_parser.py: error: unrecognized arguments: - - f i r s t - p a r a m 1 2 3 - - s e c o n d - p a r a m 4 5 6
E
======================================================================
ERROR: test_parse_cmdline (__main__.TestParser)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./test_parser.py", line 8, in test_parse_cmdline
parsed = parse_cmdline("--first-param 123 --second-param 456")
File "/home/renan/test_argparse/parser.py", line 12, in parse_cmdline
args = parser.parse_args(cmdline)
File "/usr/lib/python2.7/argparse.py", line 1691, in parse_args
self.error(msg % ' '.join(argv))
File "/usr/lib/python2.7/argparse.py", line 2361, in error
self.exit(2, _('%s: error: %s\n') % (self.prog, message))
File "/usr/lib/python2.7/argparse.py", line 2349, in exit
_sys.exit(status)
SystemExit: 2
----------------------------------------------------------------------
Ran 1 test in 0.004s
FAILED (errors=1)
As can be seen, the command line I specified (--first-param 123 --second-param 456) became
- - f i r s t - p a r a m 1 2 3 - - s e c o n d - p a r a m 4 5 6 (each character is separated by a space).
I don't understand why: what am I doing wrong?
argparse wants an "argument vector"—that is, a list of separate arguments—not a "command line" string.
And just calling split won't solve things. For example, this command line from the shell:
python script.py --first-param '123 456' --second-param 789
… will correctly give you 123 456 and 789, but this line of code:
parse_cmdline("--first-param '123 456' --second-param 789")
will not; it will give you '123 and 789, with an extra 456' that it doesn't know how to deal with.
In fact, even this is wrong:
parse_cmdline("--first-param '123' --second-param 789")
… because you'll get '123' instead of 123.
There are two ways to deal with this.
First, if you know what you're trying to pass, you can just pass it as a list instead of a string in the first place, and you don't need to worry about fiddly quoting and splitting details:
parse_cmdline(["--first-param", "123 456", "--second-param", "789"])
Alternatively, if you don't know exactly what you're trying to pass, you just know what it looks like on the shell, you can use shlex:
if cmdline is not None:
args = parser.parse_args(shlex.split(cmdline))
… and now Python will split your command line the same way as a standard Unix shell, getting the 123 456 as a single parameter.
To answer myself (I realized my mistake a few minutes later):
I am supposed to
if cmdline is not None:
args = parser.parse_args(cmdline.split())
else:
args = parser.parse_args()
Now the test passes correctly!
I am trying to use dpkt and pcapy to listen for HTTP responses on an interface
import dpkt
import pcapy
cap = pcapy.open_live('eth0',10000,1,0)
(header,payload) = cap.next()
while header:
packet = dpkt.ethernet.Ethernet(str(payload))
if str(packet.data.data.data).startswith('HTTP'):
h = dpkt.http.Response(str(packet.data.data.data))
(header,payload) = cap.next()
When I run this, it reads the first packet fine. But for the second packet, it ends up reading a wrong value of content-length. The exception is:
cona#vm-02:~$ sudo python cache-pcapy.py
Value of N 160 value of body 160
Value of N 5965717 value of body 1193
Traceback (most recent call last):
File "cache-pcapy.py", line 12, in <module>
h = dpkt.http.Response(str(packet.data.data.data))
File "/usr/local/lib/python2.7/dist-packages/dpkt/http.py", line 76, in __init__
self.unpack(args[0])
File "/usr/local/lib/python2.7/dist-packages/dpkt/http.py", line 159, in unpack
Message.unpack(self, f.read())
File "/usr/local/lib/python2.7/dist-packages/dpkt/http.py", line 90, in unpack
self.body = parse_body(f, self.headers)
File "/usr/local/lib/python2.7/dist-packages/dpkt/http.py", line 59, in parse_body
raise dpkt.NeedData('short body (missing %d bytes)' % (n - len(body)))
dpkt.dpkt.NeedData: short body (missing 5964524 bytes)
The prints for values of N and length of body are from http.py in dpkt where I added this:
elif 'content-length' in headers:
n = int(headers['content-length'])
body = f.read(n)
print 'Value of N {} value of body {}'.format(n,len(body))
if len(body) != n:
raise dpkt.NeedData('short body (missing %d bytes)' % (n - len(body)))
It seems that the wrong bytes are being read as content-length. Why does this happen?