I am trying to send data to a URL. I've a code to send it for each cpu on my Mac. But the current code loops through each of the cpustats and sends them 1 after the other. I need to send all of them in 1 POST 'cycle' but it should be formatted such that it sends it like this -
cpuStats = {nice: 123.0, idle:123.0....}
cpuStats = {nice: 123.0, idle:123.0....}
cpuStats = {nice: 123.0, idle:123.0....}
and so on...
Further, the current code pulls the stats from my Mac (with a '200 OK' for each cpustat) but when I run it on Linux, Windows, it just returns the prompt without giving any errors or stats. My guess is that it has to do with the 'break' at 'socket.error:' (My Mac has 4 cpus but the Linux and Windows machines on which I test it have 1 each.
import psutil
import socket
import time
import sample
import json
import httplib
import urllib
serverHost = sample.host
port = sample.port
thisClient = socket.gethostname()
currentTime = int(time.time())
s = socket.socket()
s.connect((serverHost,port))
cpuStats = psutil.cpu_times_percent(percpu=True)
def loop_thru_cpus():
global cpuStats
for stat in cpuStats:
stat = json.dumps(stat._asdict())
try:
command = 'put cpu.usr ' + str(currentTime) + " " + str(cpuStats[0]) + "host ="+ thisClient+ "/n"
s.sendall(command)
command = 'put cpu.nice ' + str(currentTime) + " " + str(cpuStats[1]) + "host ="+ thisClient+ "/n"
s.sendall(command)
command = 'put cpu.sys ' + str(currentTime) + " " + str(cpuStats[2]) + "host ="+ thisClient+ "/n"
s.sendall(command)
command = 'put cpu.idle ' + str(currentTime) + " " + str(cpuStats[3]) + "host ="+ thisClient+ "/n"
s.sendall(command)
params = urllib.urlencode({'cpuStats': stat, 'thisClient': 1234})
headers = headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
conn = httplib.HTTPConnection(serverHost, port)
conn.request("POST", "", params, headers)
response = conn.getresponse()
print response.status, response.reason
except IndexError:
continue
except socket.error:
print "Connection refused"
continue
print stat
loop_thru_cpus()
s.close()
If you're just trying to send the data all at once you should realize that you're not actually sending a dictionary across, but instead you're sending a string. In that case, you could easily send all the data in one go by just constructing your data like so:
data = "\n".join([json.dumps(stat._asdict()) for stat in cpuStats])
If that endpoint is someone else's that might not be sensible, but assuming that's your own endpoint you're pointing at it should be pretty trivial to unbundle this data.
Additionally I would HIGHLY suggest switching to the requests module over urllib as it extends all of the same functionality in a MUCH easier wrapper. For instance, in requests you would send that request by doing the following:
import requests
response = requests.post("your://url.here", data=data)
print response.content
Related
Here is my code, and what I am trying to do is to seperate 400 response status codes subdomains.
As you can see that I have provides three url's in subdomains list of index. Which have different status code
Now What I am trying to do is to seperate the 404 error status codes from them.
statuscode = []
statuscode.append(400)
statuscode.append(403)
statuscode.append(404)
subdomains = []
subdomains.append("https://teyrtguhigkfjn.s3.amazonaws.com/")
subdomains.append("http://google.com")
subdomains.append("https://lasdfgfldsakjas.s3.amazonaws.com/")
for x in subdomains:
url = x
try:
req = requests.get(url)
req1 = str(req.status_code) + " " + str(url) + '\n'
req2 = str(req.status_code)
req3 = str(url)
print "\n" + str(req1)
except requests.exceptions.RequestException as e:
print "Can't make the request to this Subdomain " + str(url) + '\n'
if statuscode in str(req1):
print "\nTrying to Collect the URL's whose status is 400, 400, 403"
print str(req2) + ' ' + str(req3)
But I do not succeed in that. Please see the problem, and I think I know where is the problem it is in this line of the code if statuscode in str(req1):. My Guess
Hope you get the problem.
Thank you
import requests
statuscode = [400, 403, 404]
subdomains = ["https://teyrtguhigkfjn.s3.amazonaws.com/", "http://google.com", "https://lasdfgfldsakjas.s3.amazonaws.com/"]
for url in subdomains:
try:
req = requests.get(url)
req1 = str(req.status_code) + " " + str(url) + '\n'
req2 = str(req.status_code)
print "\n" + str(req1)
if req.status_code in statuscode: #----->Update
print "\nTrying to Collect the URL's whose status is 400, 400, 403"
print str(req2) + ' ' + url
except requests.exceptions.RequestException as e:
print "Can't make the request to this Subdomain " + str(url) + '\n'
I am having trouble accessing the twitter api. It give 400 Bad request. Previously it was giving unauthorized exception. Can someone please help me figure out what am I doing wrong? I am writing this code for image upload. I thought I would first try with regular post method then would add file upload but stuck at Bad request
import oauth2 as oauth
import json
import urllib
import urllib2
from collections import namedtuple
import time
from hashlib import sha1
import hmac
import uuid
import base64
AUTHENTICATION="https://api.twitter.com/oauth2/token"
REQUEST_TOKEN="https://api.twitter.com/oauth/request_token"
AUTHORIZE_URL="https://api.twitter.com/oauth/authorize"
ACCESS_TOKEN_URL="https://api.twitter.com/oauth/access_token"
CONSUMER_KEY=""
CONSUMER_SECRET=""
ACCESS_KEY=""
ACCESS_SECRET=""
OAuth_Basic=namedtuple("OAuth_Basic", "oauth_consumer_key oauth_signature_method oauth_version")
baseString=namedtuple("baseString", "timeStamp string nonce status")
SendRequest=namedtuple("SendRequest", "method url timestamp upload")
def basestring(oauth, access_key, body=None):
timestamp = str(int(time.time()))
nonce= base64.b64encode(str(uuid.uuid4().hex))
basestring=percentencode("oauth_consumer_key") + "=" + percentencode(oauth.oauth_consumer_key) + "&"
basestring+=percentencode("oauth_nonce") + "=" + percentencode(nonce) + "&"
basestring+=percentencode("oauth_signature_method") + "=" + percentencode(oauth.oauth_signature_method) + "&"
basestring+=percentencode("oauth_timestamp") + "=" + percentencode(timestamp) + "&"
basestring+=percentencode("oauth_token") + "=" + percentencode(access_key) + "&"
basestring+=percentencode("oauth_version") + "=" + percentencode(oauth.oauth_version)
if body:
basestring+="&" + percentencode("status") + "=" + percentencode(body)
return baseString(timeStamp=timestamp, string=basestring, nonce=nonce, status=body)
def sign_request(signature, consumer_secret, access_secret):
key = consumer_secret + "&" + access_secret
hashed = hmac.new(key, signature, sha1)
return hashed.digest().encode("base64").rstrip('\n')
def signaturebasestring(request, basestr, upload=False):
if not upload:
return percentencode(request.method) + "&" + percentencode(request.url) + "&" + percentencode(basestr)
return percentencode(request.url) + "&" + percentencode(basestr)
def sendrequest(s_oauth, s_basestring, s_request, s_signature):
oauth_header="OAuth oauth_consumer_key=\""+s_oauth.oauth_consumer_key+"\", oauth_nonce=\""+s_basestring.nonce+"\", oauth_signature=\""+percentencode(s_signature)+"\", oauth_signature_method=\""+s_oauth.oauth_signature_method+"\", oauth_timestamp=\""+s_request.timestamp+"\", oauth_token=\""+ACCESS_KEY+"\", oauth_version=\""+s_oauth.oauth_version+"\""
headers = {"Authorization": oauth_header, "Content-type": "application/x-www-form-urlencoded", "Host":"api.twitter.com", "User-Agent":"OAuth gem v0.4.4", "oauth_consumer_key":s_oauth.oauth_consumer_key}
status = {"status":s_basestring.status}
encoded_args = urllib.urlencode(status)
req = urllib2.Request(s_request.url, encoded_args, headers)
try:
resp = urllib2.urlopen(req)
print resp
except urllib2.HTTPError, e:
print "httperror :" + str(e.code)
print e.reason
except urllib2.URLError, e:
print "urlerror :" + str(e.code)
print e.reason
except httplib.HTTPException, e:
print "httpexception :" + str(e.code)
print e.reason
def percentencode(val):
return val.replace(" ", "%20").replace("!", "%21").replace("&", "%26").replace("/", "%2F").replace("=", "%3D").replace("+", "%2B").replace(",", "%2C").replace("-", "%2D").replace(".", "%2E")
p_oauth=OAuth_Basic(oauth_consumer_key=CONSUMER_KEY, oauth_signature_method="HMAC-SHA1", oauth_version="1.0")
p_basestring=basestring(p_oauth, ACCESS_KEY, "hello world")
p_request = SendRequest(method="POST", url="https://api.twitter.com/1.1/statuses/update.json", timestamp=p_basestring.timeStamp, upload=False)
signature=sign_request(signaturebasestring(p_request, p_basestring.string, False), CONSUMER_SECRET, ACCESS_SECRET)
sendrequest(p_oauth, p_basestring, p_request, signature)
I found the error. The percent encoding was happening in lower case, Changed to upper case and everything started working. There were couple of site that helped validate the OAuth Header.
Thanks
I wanted to create a Python program that does several things. Ping all addresses in a predefined network, gather the DNS information, write a file with IP address, DNS name, ping fail or pass, date. Then run and email the resulting file to myself once a week, every Friday. I have created this program and will post my own answer. I am new to Python and was able to get this written with the help from other answers posted on this site. Thanks to all those who contributed answers on this site. Hope the answer I post will help someone else.
#!/usr/bin/python3.4
#Above statement makes sure you are using version 3.4
#when multiple versions are installed. has to be the 1st line.
# Import modules
import subprocess
import socket
import errno
import time
import datetime
import ipaddress
today = datetime.date.today()
# define DNS lookup and error handling
# return none,none,none needed otherwise if no DNS record
# the routine errors out and the program stops
def lookup(addr):
try:
return socket.gethostbyaddr(addr)
except socket.herror:
return None, None, None
# Prompt the user to input a network address
# commented out the prompt for input so it can run unattended
# net_addr = input("Enter a network address in CIDR
format(ex.192.168.1.0/24): ")
net_addr = ('192.168.1.0/24')
# Create the network
ip_net = ipaddress.ip_network(net_addr)
# Get all hosts on that network
all_hosts = list(ip_net.hosts())
# Configure subprocess to hide the console window
# removed code due to errors not windows linux
# setup online and offline count variables
offCnt = 0
onCnt = 0
# Open file and or create if it doesn't exist.
# file to be overwritten each time the program is run.
file = open("lab-ip.doc","w")
# For each IP address in the subnet,
# run the ping command with subprocess.popen interface
# Grab the DNS information for each IP address
# Print to console add counters and write to file.
for i in range(len(all_hosts)):
output = subprocess.Popen(['ping', '-c', '2', str(all_hosts[i])],
stdout=subprocess.PIPE).communicate()[0]
name,alias,addresslist = lookup(str(all_hosts[i]))
if "Destination Host Unreachable" in output.decode('utf-8'):
print(str(all_hosts[i]), " Ping Fail", str(name), today)
file.write(str(all_hosts[i]) + " Ping Fail - " + str(name) + " " + str(today) + "\n")
offCnt = offCnt + 1
elif "Request timed out" in output.decode('utf-8'):
print(str(all_hosts[i]), " Ping Fail", str(name), today)
file.write(str(all_hosts[i]) + " Ping Fail - " + str(name) + " " + str(today) + "\n")
offCnt = offCnt + 1
else:
print(str(all_hosts[i]), " Ping Pass", str(name), today)
file.write(str(all_hosts[i]) + " Ping Pass - " + str(name) + " " + str(today) + "\n")
onCnt = onCnt + 1
print ("Pass count = ", str(onCnt))
file.write("Pass count = " + str(onCnt))
print ("Fail count = ", str(offCnt))
file.write(" Fail count = " + str(offCnt))
file.close()
# Import yagmail for the actual sending function
import yagmail
yag = yagmail.SMTP('Gmail-id', 'gmail-pswd')
yag.send('email#email.com', subject = "Lab-ip List",contents = 'lab-ip.doc')
yag.send('email2#email2.com', subject = "Lab-ip List",contents = 'lab-ip.doc')
#end
After multiple attempts, this code still fails. What I am trying to do is send "cpu stats" as JSON over to the server. The thing is, cpustats alone is fine - there is just 1 namedtuple with the different cpupercentages - user, idle, etc.. But 'percpu' returns a list of each cpu's namedtuple (of user, idle, etc.). So I cannot convert the list to a dictionary. I am trying to loop through the list and then send each one of the namedtuple to the server. (For ref. - I am using 2.7.5). The script worked fine without the attempt to loop and try/except - it returned a '200 OK'. But now when I run it it doesn't even return an error, any response message/status. It's as if the script is just bypassing the whole try/except block. Just the 'print cpuStats' line at the end delivers as it should. (The indentation in this question is a bit off but it's fine in the script)
import psutil
import socket
import time
import sample
import json
import httplib
import urllib
serverHost = sample.host
port = sample.port
thisClient = socket.gethostname()
currentTime = int(time.time())
s = socket.socket()
s.connect((serverHost,port))
cpuStats = psutil.cpu_times_percent(percpu=True)
print cpuStats
def loop_thru_cpus():
for i in cpuStats:
cpuStats = cpuStats[i]
cpuStats = json.dumps(cpuStats._asdict())
try:
command = 'put cpu.usr ' + str(currentTime) + " " + str(cpuStats[0]) + "host ="+ thisClient+ "/n"
s.sendall(command)
command = 'put cpu.nice ' + str(currentTime) + " " + str(cpuStats[1]) + "host ="+ thisClient+ "/n"
s.sendall(command)
command = 'put cpu.sys ' + str(currentTime) + " " + str(cpuStats[2]) + "host ="+ thisClient+ "/n"
s.sendall(command)
command = 'put cpu.idle ' + str(currentTime) + " " + str(cpuStats[3]) + "host ="+ thisClient+ "/n"
s.sendall(command)
params = urllib.urlencode({'cpuStats': cpuStats, 'thisClient': 1234})
headers = httplib.HTTPConnection(serverHost, port)
conn.request("POST", "", params, headers)
response = conn.response()
print response.status, response.reason
except IndexError:
break
i = i+1
s.close()
Instead of:
def loop_thru_cpus():
for i in cpuStats:
cpuStats = cpuStats[i]
cpuStats = json.dumps(cpuStats._asdict())
...
i = i+1
Try:
def loop_thru_cpus():
for stat in cpuStats:
stat = json.dumps(stat._asdict())
When you say
for i in cpuStats:
i takes on values from cpuStats. i is not an integer in this case. So i = i+1 makes no sense.
cpuStats = cpuStats[i]
This probably raised an IndexError (since i is not an integer), but for some reason you are not seeing the exception that was raised.
Also note that you are redefining cpuStats here, which is probably not what you want to do.
You probably do have indentation errors in your code. Running the code you posted through cat -A shows tabs (indicated below by ^I):
try:$
^I $
command = 'put cpu.usr ' + str(currentTime) + " " + str(cpuStats[0]) + "host ="+ thisClient+ "/n"$
...
params = urllib.urlencode({'cpuStats': cpuStats, 'thisClient': 1234})$
...
^I print response.status, response.reason$
$
^I except IndexError:$
break$
You can not mix tabs and spaces an indentation in Python code. Either use one or the other. The PEP8 style guide (and most code you see on the net) uses 4 spaces. Mixing tabs and spaces usually results in an IndentationError, but sometimes you don't get an error and just code that behaves in unexpected ways. So (if using the 4-spaces convention) be careful to use an editor that adds 4 spaces when the tab key is pressed.
Since you are not seeing the IndexError, you may not be seeing the IndentationError that should have been occurred either. How exactly are you running the problem?
I have a problem.
I've written this part of code in Python (Tornado web server):
if command == 'RESTARTNWK':
op_group = "A3"
op_code = "E0"
netid = hextransform(int(nid), 16)
packet_meta = "*%s;%s;%s;#"
pkt_len = hextransform(0, 2)
packet = packet_meta % (op_group, op_code, pkt_len)
packet = packet.upper()
op_group_hex=0xA3
op_code_hex=0xE0
cmdjson = packet2json(op_group_hex,op_code_hex, packet)
mynet_type="ztc"
print("\t\t " + packet + "\n")
#TODO : -write command into db
ts = datetime.datetime.now().isoformat()
mynet_type ="ztc"
self.lock_tables("write", ['confcommands'])
self.db.execute("INSERT INTO confcommands (network_id, ntype, timestamp, command) \
VALUES (%s,%s,%s,%s)", nid, mynet_type, ts, cmdjson)
self.unlock_tables();
# TODO: - open the /tmp/iztc file in append mode
cmdfile = open('/tmp/iztc', 'a')
# - acquire a lock "only for the DB case, it's easier"
# - write the packet
cmdfile.write(netid + "\t"+ mynet_type + "\t"+ ts + "\t"+ cmdjson +"\n");
# - release the lock "only for the DB case, it's easier"
# - close the file
cmdfile.close()
if command == 'RESTARTNWK':
opcodegroupr = "A4"
opcoder = "E0"
#Code for retrieving the MAC address of the node
como_url = "".join(['http://', options.como_address, ':', options.como_port,
'/', ztc_config, '?netid=', netid,
'&opcode_group=', opcodegroupr,
'&opcode=', opcoder, '&start=-5m&end=-1s'])
http_client = AsyncHTTPClient()
response = yield tornado.gen.Task(http_client.fetch, como_url)
ret = {}
if response.error:
ret['error'] = 'Error while retrieving unregistered sensors'
else:
for line in response.body.split("\n"):
if line != "":
value = int(line.split(" ")[6])
ret['response'] = value
self.write(tornado.escape.json_encode(ret))
self.finish()
In this code I receive the restart network command from the user. After some settings, I write the relative command in a db table named confcommands. The server will read this command and will send to the specified network the restart signal.
After this, if all it's ok, the network resend me the response. I read this response with a http request to my server (como), and wait for the asynchronous response.
Where the response is written by the network, I have to find this in the packet. The value response is the sixth element. Other information of the packet are the opgroup and opcode, the network from which is the response, and other informations.
Then I write the response for the user.
I don't know if this code is right... can work this? The structure seems to me right....
Thank you all for any suggestions!