Develop Reference

writing variables in pandas dataframe with header

i have a script that runs api calls agains a flask app. i want to create a pandas datafram with the statuscode and the elapsed time of the request which i can write to a csv file.
My problem is that only one entry ends up being in the csv file and i dont know why.
the headers should be "statuscode" and "elapsed time".
when i am printing the statuscode and elapsedtime variables every response is printed and not only one
with this csv file i want to create a grap to visualize the responstimes
i tried to write the "write_df" fuction but ended up using the variables from the requests in the "send_api_request" function.
import requests
import datetime
import concurrent.futures
import csv
import pandas as pd
HOST = 'http://127.0.0.1:5000'
API_PATH = '/'
ENDPOINT = HOST + API_PATH
MAX_THREADS = 8
CONCURRENT_THREADS = 10
csv_path = "flasktests.csv"
try:
file = open(csv_path, 'w', newline='')
writer = csv.writer(file)
except:
print("error opening or writing to the CSV file!")
def send_api_request():
try:
#print ('Sending API request: ', ENDPOINT)
r = requests.get(ENDPOINT)
if r.status_code == 200:
#print('Received: ', r.status_code, r.elapsed)
responses = {"statuscode":[r.status_code], "elapsed time": [r.elapsed]}
statuscode = r.status_code
elapsedtime = r.elapsed
print(statuscode, elapsedtime)
df = pd.DataFrame([statuscode,elapsedtime], columns=["statuscode","elapsed time"])
df.to_csv(csv_path, index=False)
elif r.status_code == 417:
print('Received error code:', r.status_code, r.json())
except Exception as e:
print("error",str(e))
def write_df(statuscode, elapsedtime):
print(statuscode,elapsedtime)
df = pd.DataFrame({"statuscode":[statuscode], "elapsed time": [elapsedtime]})
df.to_csv(csv_path, index=False)
print(df)
with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
futures = [executor.submit(send_api_request) for x in range (CONCURRENT_THREADS)]
executor.shutdown(wait=True)
any ideas what i am doing wrong here?
Thank you!

df = pd.DataFrame([statuscode,elapsedtime], columns=["statuscode","elapsed, "elapsed time": [elapsedtime]})
df.to_csv(csv_path, index=False)
Here, you are writing a csv with only one entry, because your dataframe df only has one row.
And everytime this function is being executed, you will overwrite the existing csv. This is why the output file only contains one line (2 if you count the header, which should be there)
I see you are trying to use multithreading, which is probably not needed. If you do want to keep using it, you will need to make sure that you are not writing to the file with two threads at the same time, which is a lot of overhead. Instead, I would suggest you do something like this:
def send_api_request():
try:
#print ('Sending API request: ', ENDPOINT)
r = requests.get(ENDPOINT)
if r.status_code == 200:
#print('Received: ', r.status_code, r.elapsed)
return r.status_code, r.elapsed
elif r.status_code == 417:
print('Received error code:', r.status_code, r.json())
return r.status_code, None
except Exception as e:
print("error",str(e))
# number of times you want to call the API
nb_api_calls = 10
codes = []
elapsed_times = []
for i in range(nb_api_calls):
code, elapsed_time = send_api_request()
if elapsed_time is None:
# An error happened, choose what do do with that information
# Here I am just skipping it
pass
else:
codes.append(code)
elapsed_times.append(elapsed_time)
df = pd.DataFrame({"statuscode": codes,"elapsed time": elapsed_times})
df.to_csv(csv_path, index=False)
Note that, with the current configuration, the "statuscode" column will only contain the value 200 many times

Python Sockets: my recv protocol receive all the data when i debug it but not when i run it doesnt

So my project is that I need to send a jpg image from one computer to another computer in the same network. To send the data I split the data into chunks of at least 9999 bytes and then I create a length header that tells the length of the data and I attach it to the start of the massage. here is the code:
the protocol:
import os.path
LENGTH_FIELD_SIZE = 4
PORT = 8820
COMANDS_LIST = "TAKE_SCREENSHOT\nSEND_PHOTO\nDIR\nDELETE\nCOPY\nEXECUTE\nEXIT".split("\n")
def check_cmd(data):
"""
Check if the command is defined in the protocol, including all parameters
For example, DELETE c:\work\file.txt is good, but DELETE alone is not
"""
command = ""
file_location =""
splited_data = data.split(maxsplit=1)
if len(splited_data) == 2:
command, file_location = splited_data
return (command in COMANDS_LIST) and (file_location is not None)
elif len(splited_data) == 1:
command = splited_data[0]
return command in ["TAKE_SCREENSHOT","EXIT","SEND_PHOTO"]
return False
# (3)
def create_msg(data):
"""
Create a valid protocol message, with length field
"""
data_len = len(str(data))
if data_len > 9999 or data_len == 0:
print(f"data len is bigger then 9999 or is 0, data len = {data_len} ")
return False
len_field = str(data_len).zfill(4)
# (4)
print(len_field)
return True ,f"{len_field}{data}"
def get_msg(my_socket):
"""
Extract message from protocol, without the length field
If length field does not include a number, returns False, "Error"
"""
lenght_field = ""
data = ""
try:
while len(lenght_field) < 4:
lenght_field += my_socket.recv(4).decode()
except RuntimeError as exc_run:
return False, "header wasnt sent properly"
if not lenght_field.isdigit():
return False, "error, length header is not valid"
lenght_field = lenght_field.lstrip("0")
while len(data) < int(lenght_field):
data += my_socket.recv(int(lenght_field)).decode()
return True, data
now the protocol works fine when I use the same computer for both server and client and when I debug get_msg on the other computer. when I'm not, it seems that the problem is that the part that recv the header will recv something else after a few successful recv and return an error message.
here are the server parts:
import socket
import pyautogui as pyautogui
import protocol
import glob
import os.path
import shutil
import subprocess
import base64
IP = "0.0.0.0"
PORT = 8820
PHOTO_PATH = r"C:\Users\Innon\Pictures\Screenshots\screenShot.jpg"# The path + filename where the screenshot at the server should be saved
def check_client_request(cmd):
"""
Break cmd to command and parameters
Check if the command and params are good.
For example, the filename to be copied actually exists
Returns:
valid: True/False
command: The requested cmd (ex. "DIR")
params: List of the cmd params (ex. ["c:\\cyber"])
"""
# Use protocol.check_cmd first
cmd_arr = cmd.split(maxsplit=1)
command = cmd_arr[0]
file_location = None
if len(cmd_arr) == 2:
file_location = cmd_arr[1]
if file_location == None:
return protocol.check_cmd(cmd) ,command, file_location
else:
file_location = tuple(str(file_location).split())
if (os.path.exists(file_location[0])):
return protocol.check_cmd(cmd) , command , file_location
return False , command , file_location
# Then make sure the params are valid
# (6)
def handle_client_request(command,params):
"""Create the response to the client, given the command is legal and params are OK
For example, return the list of filenames in a directory
Note: in case of SEND_PHOTO, only the length of the file will be sent
Returns:
response: the requested data
"""
# (7)
response = "no server response"
if command == "DIR":
response = glob.glob(f"{params[0]}\\*.*" )
if command == "DELETE":
os.remove(params[0])
response = f"{params[0]} was deleted"
if command == "COPY":
try:
shutil.copy(params[0],params[1])
response = f"{params[0]} was copyed to {params[1]}"
except FileNotFoundError as ex1:
response = ex1
except IndexError as ex2:
response = ex2
if command == "EXECUTE":
subprocess.call(params[0])
response = f"{params[0]} was executed"
if command == "TAKE_SCREENSHOT":
#todo find a way to know and create the locatipn of screen shot to be saved
myScreenshot = pyautogui.screenshot()
myScreenshot.save(PHOTO_PATH)
response = f"screen shot have been taken and been saved at {PHOTO_PATH}"
if command == "SEND_PHOTO":
with open(PHOTO_PATH, "rb") as file:
file_data = base64.b64encode(file.read()).decode()
print(file_data)
is_vaild_response, img_length = protocol.create_msg(len(file_data))
print(img_length)
img_data = ""
if not is_vaild_response:
response = "img length data isnt valid"
return response
while len(file_data) > 0:
chunk_data = file_data[:9999]
is_vaild_response, data = protocol.create_msg(chunk_data)
if not is_vaild_response:
response = "img data isnt valid"
return response
img_data += data
file_data = file_data[9999:]
response = f"{img_length}{img_data}"
return response
def main():
# open socket with client
server_socket = socket.socket()
server_socket.bind((IP,PORT))
server_socket.listen(1)
# (1)
client_socket, addr = server_socket.accept()
# handle requests until user asks to exit
while True:
# Check if protocol is OK, e.g. length field OK
valid_protocol, cmd = protocol.get_msg(client_socket)
print(f"got message {valid_protocol}")
if valid_protocol:
# Check if params are good, e.g. correct number of params, file name exists
valid_cmd, command, params = check_client_request(cmd)
print(f"check_client_request {valid_cmd}")
if valid_cmd:
# (6)
if command == 'EXIT':
break
if command == 'SEND_PHOTO':
data = handle_client_request(command, params)
client_socket.sendall(data.encode())
continue
# prepare a response using "handle_client_request"
data = handle_client_request(command,params)
# add length field using "create_msg"
is_vaild_response , response = protocol.create_msg(data)
print(f"creat_msg {is_vaild_response}")
# send to client
if is_vaild_response:
client_socket.sendall(response.encode())
else:
# prepare proper error to client
resp = 'Bad command or parameters'
is_vaild_response , response = protocol.create_msg(resp)
# send to client
client_socket.sendall(response.encode())
else:
# prepare proper error to client
resp = 'Packet not according to protocol'
is_vaild_response, response = protocol.create_msg(resp)
#send to client
client_socket.sendall(response.encode())
# Attempt to clean garbage from socket
client_socket.recv(1024)
# close sockets
resp = "Closing connection"
print(resp)
is_vaild_response, response = protocol.create_msg(resp)
client_socket.sendall(response.encode())
client_socket.close()
server_socket.close()
if __name__ == '__main__':
main()
and the client:
import socket
import base64
import protocol
IP = "127.0.0.1"
SAVED_PHOTO_LOCATION = r'C:\Users\Innon\Pictures\Saved Pictures\screenShot.jpg' # The path + filename where the copy of the screenshot at the client should be saved
def handle_server_response(my_socket, cmd):
"""
Receive the response from the server and handle it, according to the request
For example, DIR should result in printing the contents to the screen,
Note- special attention should be given to SEND_PHOTO as it requires and extra receive
"""
# (8) treat all responses except SEND_PHOTO
if "SEND_PHOTO" not in cmd:
vaild_data, data = protocol.get_msg(my_socket)
if vaild_data:
return data
# (10) treat SEND_PHOTO
else:
pic_data = ""
vaild_pick_len, pic_len = protocol.get_msg(my_socket)
if pic_len.isdigit() == False:
print(f"picture length is not valid. got massage: {pic_len}")
return
with open(SAVED_PHOTO_LOCATION, "wb") as file:
while len(pic_data) < int(pic_len):
vaild_data, data = protocol.get_msg(my_socket)
if not vaild_data:
return f"img data isnt valid. {data}"
pic_data += data
print(pic_data)
file.write(base64.b64decode(pic_data.encode()))
return "img was recived succesfully "
def main():
# open socket with the server
my_socket = socket.socket()
my_socket.connect((IP,8820))
# (2)
# print instructions
print('Welcome to remote computer application. Available commands are:\n')
print('TAKE_SCREENSHOT\nSEND_PHOTO\nDIR\nDELETE\nCOPY\nEXECUTE\nEXIT')
# loop until user requested to exit
while True:
cmd = input("Please enter command:\n")
if protocol.check_cmd(cmd):
valid_pack , packet = protocol.create_msg(cmd)
if valid_pack:
my_socket.sendall(packet.encode())
print(handle_server_response(my_socket, cmd))
if cmd == 'EXIT':
break
else:
print("Not a valid command, or missing parameters\n")
my_socket.close()
if __name__ == '__main__':
main()
here is how the problem looks like:thi is how it looks
here is how to needs look like:
the right way
thank you.

the solution was to change get_msg function in the protocol:
while len(data) < int(lenght_field):
data += my_socket.recv(int(lenght_field) - len(data)).decode()
instead of:
while len(data) < int(lenght_field):
data += my_socket.recv(int(lenght_field)).decode()

how to check if cpanel can login successfully with python2.7?

i need a script to make it like a cpanel checker, with more than 1 url and the url is stored in a txt file.
usage : python script.py list.txt
format in file list.txt : https://demo.cpanel.net:2083|democom|DemoCoA5620
this is my code but it doesn't work, can someone help me?
Thanks.
import requests, sys
from multiprocessing.dummy import Pool as ThreadPool
try:
with open(sys.argv[1], 'r') as f:
list_data = [line.strip() for line in f if line.strip()]
except IOError:
pass
def cpanel(url):
try:
data = {'user':'democom', 'pass':'DemoCoA5620'}
r = requests.post(url, data=data)
if r.status_code==200:
print "login success"
else:
print "login failed"
except:
pass
def chekers(url):
try:
cpanel(url)
except:
pass
def Main():
try:
start = timer()
pp = ThreadPool(25)
pr = pp.map(chekers, list_data)
print('Time: ' + str(timer() - start) + ' seconds')
except:
pass
if __name__ == '__main__':
Main()

I fixed your code in a way that it will return an actual array containing a boolean array indicating the success of the cpanel function.
from __future__ import print_function
import requests
from multiprocessing.pool import ThreadPool
try:
list_data = ["https://demo.cpanel.net:2083|democom|DemoCoA5620",
"https://demo.cpanel.net:2083|UserDoesNotExist|WRONGPASSWORD",
]
except IOError:
pass
def cpanel(url):
try:
# try to split that url to get username / password
try:
url, username, password = url.split('|')
except Exception as e:
print("Url {} seems to have wrong format. Concrete error: {}".format(url, e))
return False
# build the correct url
url += '/login/?login_only=1'
# build post parameters
params = {'user': username,
'pass': password}
# make request
r = requests.post(url, params)
if r.status_code==200:
print("login for user {} success".format(username))
return True
else:
print("login for user {} failed due to Status Code {} and message \"{}\"".format(username, r.status_code, r.reason))
return False
except Exception as e:
print("Error occured for url {} ".format(e))
return False
def chekers(url):
return cpanel(url)
def Main():
try:
# start = timer()
pp = ThreadPool(1)
pr = pp.map(chekers, list_data)
print(pr)
# print('Time: ' + str(timer() - start) + ' seconds')
except:
pass
if __name__ == '__main__':
Main()
Output:
login for user democom success
login for user UserDoesNotExist failed due to Status Code 401 and message "Access Denied"
[True, False]
Be aware that I replaced your file read operation by some fixed urls.
Since you use request.post I guess you actually want to POST something to that urls. Your code does not do that. If you just want to send a request, use the requests.get method.
See the official documentation for the requests packet: https://2.python-requests.org/en/master/user/quickstart/#make-a-request for more details.
Also note that
"but it doesn't work"
is NOT a question.

Python download videos from HTTP URL with bad internet

I have problem with downloading videos from my server e.g. http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Cornaro%20USLUGE.mp4
All works perfectly when internet is OK, but when I disconnect LAN cable from Raspberry Pi and stay like that less than 10-15 seconds. But when internet is off more than 10-15 seconds, my download does not continue or videos are not properly downloaded (I merge them later with MP4Box and they need to be). If someone has suggestion how to solve this problem and help me I would appreciate it very much.
Here is my code:
import os
import urllib
import urllib2
import time
import commands
import requests
import shutil
from urllib2 import URLError
urls = ['http://screensfiles.dbtouch.com/screens2/Companies/89/HD/00 APPS OVERVIEW.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Cornaro USLUGE.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/ILIRIJA BIOGRAD 2016.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Restoran marina.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/HT Screens.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Hotels Touch - Tasks.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Croatia Full of life.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/04 PROJECTS.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/05 ATTEND.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Cornaro Hotel.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Plurato dron snimka 2.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Plurato dron snimka 2.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Plurato dron snimka 2.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Cornaro USLUGE.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Cornaro USLUGE.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Hotels Touch - Screens.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Hotels Touch - Screens.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Hotels Touch - Tasks.mp4',
'http://screensfiles.dbtouch.com/screens2/Companies/89/HD/Hotels Touch - Screens.mp4']
directory = "/home/pi/pythonSignage/current_playlist/videos_to_merge/"
i=1
for url in urls:
i += 1
print("current iter: ")
print(i)
if (len(urls) > 1):
url_formatted = url.split('/')[-1].replace(" ", "").replace("%20", "") + " "
else:
url_formatted = url.split('/')[-1].replace(" ", "").replace("%20", "")
url_formatted_name = url.split('/')[-1].replace(" ", "").replace("%20", "").rstrip()
while True:
print("inside while true")
try:
""" method 0 doesn't work """
print("try")
response = urllib2.urlopen(url, timeout=5)
content = response.read()
print("content")
f = open(directory + url_formatted_name, 'wb')
f.write(content)
f.close()
""" method 1 doesn't work """
#video_fancy_downloader = urllib.FancyURLopener()
#video_fancy_downloader.retrieve(url, directory + url_formatted_name)
""" method 2 - doesn't work """
#my_file = urllib.URLopener()
#my_file = retrieve(url, directory + url_formatted_name)
""" method 3 - doesn't work """
#response = requests.get(url, stream=True)
#response.raise_for_status()
#with open(directory + url_formatted_name, 'wb') as handle:
# for block in response.iter_content(1024):
# handle.write(block)
except:
print("error download, sleep 5 sec")
time.sleep(5)
print("end")

I have managed to solve my problem. Maybe this is not best approach but it works.
Here is function for downloading video and returns response:
def do_download(destination, url):
comm = ["wget", "-c", "-O", destination, "-t", "15000", "-T", "5", url]
proc = subprocess.Popen(comm, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
tmp = proc.stdout.read()
if "wget: unable to resolve host address" in tmp:
return False
else:
return True
core part of downloading function is almost the same, but now it calls do_download inside while loop and checks for response:
if os.path.isfile(directory+url_formatted_name) is False:
print("must download file!")
while downl_success is False:
print("inside while true")
try:
print("try")
while(do_download(directory + url_formatted_name, url) is False):
print(" ------- inside while for download ----------- ")
time.sleep(5)
downl_success = True
print("file downloaded fully!")
break
except HTTPError, e:
print "HTTPError", e.code, url
time.sleep(5)
except URLError, e:
print "URL Error", e.reason, url
time.sleep(5)
else:
print("file already downloaded no need to download it again!")

Retrieving response from network and print with Python

I have a problem.
I've written this part of code in Python (Tornado web server):
if command == 'RESTARTNWK':
op_group = "A3"
op_code = "E0"
netid = hextransform(int(nid), 16)
packet_meta = "*%s;%s;%s;#"
pkt_len = hextransform(0, 2)
packet = packet_meta % (op_group, op_code, pkt_len)
packet = packet.upper()
op_group_hex=0xA3
op_code_hex=0xE0
cmdjson = packet2json(op_group_hex,op_code_hex, packet)
mynet_type="ztc"
print("\t\t " + packet + "\n")
#TODO : -write command into db
ts = datetime.datetime.now().isoformat()
mynet_type ="ztc"
self.lock_tables("write", ['confcommands'])
self.db.execute("INSERT INTO confcommands (network_id, ntype, timestamp, command) \
VALUES (%s,%s,%s,%s)", nid, mynet_type, ts, cmdjson)
self.unlock_tables();
# TODO: - open the /tmp/iztc file in append mode
cmdfile = open('/tmp/iztc', 'a')
# - acquire a lock "only for the DB case, it's easier"
# - write the packet
cmdfile.write(netid + "\t"+ mynet_type + "\t"+ ts + "\t"+ cmdjson +"\n");
# - release the lock "only for the DB case, it's easier"
# - close the file
cmdfile.close()
if command == 'RESTARTNWK':
opcodegroupr = "A4"
opcoder = "E0"
#Code for retrieving the MAC address of the node
como_url = "".join(['http://', options.como_address, ':', options.como_port,
'/', ztc_config, '?netid=', netid,
'&opcode_group=', opcodegroupr,
'&opcode=', opcoder, '&start=-5m&end=-1s'])
http_client = AsyncHTTPClient()
response = yield tornado.gen.Task(http_client.fetch, como_url)
ret = {}
if response.error:
ret['error'] = 'Error while retrieving unregistered sensors'
else:
for line in response.body.split("\n"):
if line != "":
value = int(line.split(" ")[6])
ret['response'] = value
self.write(tornado.escape.json_encode(ret))
self.finish()
In this code I receive the restart network command from the user. After some settings, I write the relative command in a db table named confcommands. The server will read this command and will send to the specified network the restart signal.
After this, if all it's ok, the network resend me the response. I read this response with a http request to my server (como), and wait for the asynchronous response.
Where the response is written by the network, I have to find this in the packet. The value response is the sixth element. Other information of the packet are the opgroup and opcode, the network from which is the response, and other informations.
Then I write the response for the user.
I don't know if this code is right... can work this? The structure seems to me right....
Thank you all for any suggestions!