I use a script with an infinite loop to upload sensor data to parse.com. I use
batcher.batch_save(myDataPoints)
to upload the data.
The problem is if the computer where the script runs (a raspberry pi) looses the internet connection, the scipt will exit on an error because the batcher can't reach the parse api.
how can I avoid this? If there is no internet connection, I would like the program to execute some code and keep looping, but NOT to exit on an error.
thanks.
If you just want to keep going after you get an error you can just use try...except, e.g.
while True:
try:
upload_function()
except Exception:
pass
This should catch your exception and instead of exiting out of your while loop, just continue on. You can put whatever you want to do there, you don't have to just leave it as pass.
def spinner(newList):
count = 0
while connection_is_bad:
count +=1
x= uploader(newList)
If your data is a list I think it gets easier
def uploader(someList):
last_value = ''
while connection_is_good:
for item in someList:
do some uploading
last_value = item
return something
newList = someList[someList.index(last_value) + 1:]
x = spinner(newList)
Here is my thought about some pseudo-code. I can imagine a number of ways to keep track of the data being transferred but not completely sure until I saw how it was collected
Related
I have a python script driving me crazy.
Right now i have a infinite loop that checks a data and get into a function in a specific situation.
while true:
time.sleep(1)
peso = getPeso()
if peso > 900:
processo = multiprocessing.Process(target=processo_automatico)
processo.start()
processo.join()
andare() #this is where it should go after the process.
The function works a lot of times but nondeterministically it hangs at the end of the process (literally finish and hangs there forever. How do i know this? i tried with logs).
So at the beginning I tried to terminate it with exit codes:
processo.join(timeout=10)
if processo.exitcode is None:
errore_cicalino() #this is just a warning for me
processo.kill()
elif processo.exitcode != 0:
errore_cicalino()
processo.kill()
but this never worked. NEVER.
So i tried without the join(). Tried with is_alive().
time.sleep(10)
if processo.is_alive():
processo.terminate()
errore_cicalino()
and even like this, it never entered this if.
This is driving me crazy, i accept the fact that the process could fail but, after the timeout, i should be able to terminate it and carry on with the script.
The script is running on a Raspberry Pi 4 2 GB.
Any idea?
Minimal example:
while True:
time.sleep(10)
processo = multiprocessing.Process(target=processo_automatico)
processo.start()
processo.join()
code randomly hangs at the end of the started process and cannot terminate in any way.
processo_automatico() is a function where the script get a picture from a camera and upload it in a DB thanks to another module.
def processo_automatico():
now = str(datetime.now())
now = now.replace(":", "-")
foto = CAMERA.read_camera()
cv2.imwrite("/var/www/html/" + now + ".png", foto)
DB.insert("/var/www/html/" + now + ".png", peso)
They don't create exceptions and i already tried to add to the end of the function a log info(executed even when the code hangs)
Solved.
What was that?
Well, my focus was on the end of the process while the issue was right at the start.
See the function getPeso()? That one just gets values from the serial.
After a few hours of getting those values the raspberry py just starts to see random values and you gotta reboot to fix it.
I wasn't prepared for that. My function just became a recursive infinite function with no break.
My tip, do not use infinite cycles except if you really need to or at least think if it could get stuck somewhere and check it.
I am relatively new to Python, and programming as a whole. I am progressively getting the hang of it, however I have been stumped as of late in regards to one of my latest projects. I have a set of Atlas Scientific EZO circuits w/ their corresponding sensors hooked up to my Raspberry Pi 3. I can run the i2c script fine, and the majority of the code makes sense to me. However, I would like to pull data from the sensors and log it with a time stamp in a CSV file, taking data points in timed intervals. I am not quite sure how to pull the data from the sensor, and put it into a CSV. Making CSVs in Python is fairly simple, as is filling them with data, but I cannot seem to understand how I would make the data that goes into the CSV the same as what is displayed in the terminal when one runs the Poll function. Attached is the i2c sample code from Atlas' website. I have annotated it a bit more so as to help me understand it better.
I have already attempted to make sense of the poll function, but am confused in regards to the self.file_write and self.file_read methods used throughout the code. I do believe they would be of use in this instance but I am generally stumped in terms of implementation. Below you will find a link to the Python script (i2c.py) written by Atlas Scientific
https://github.com/AtlasScientific/Raspberry-Pi-sample-code/blob/master/i2c.py
I'm guessing by "the polling function" you are referring to this section of the code:
# continuous polling command automatically polls the board
elif user_cmd.upper().startswith("POLL"):
delaytime = float(string.split(user_cmd, ',')[1])
# check for polling time being too short, change it to the minimum timeout if too short
if delaytime < AtlasI2C.long_timeout:
print("Polling time is shorter than timeout, setting polling time to %0.2f" % AtlasI2C.long_timeout)
delaytime = AtlasI2C.long_timeout
# get the information of the board you're polling
info = string.split(device.query("I"), ",")[1]
print("Polling %s sensor every %0.2f seconds, press ctrl-c to stop polling" % (info, delaytime))
try:
while True:
print(device.query("R"))
time.sleep(delaytime - AtlasI2C.long_timeout)
except KeyboardInterrupt: # catches the ctrl-c command, which breaks the loop above
print("Continuous polling stopped")
If this is the case then if looks like you can recycle most of this code for your use. You can grab the string you are seeing in your console with device.query("R"), instead of printing it, grab the return value and write it to your CSV.
I think You should add method to AtlasI2C class which will write data to file
Just type under AtlasI2C init() this method:
def update_file(self, new_data):
with open(self.csv_file, 'a') as data_file:
try:
data = "{}\n".format(str(new_data))
data_file.write(data)
except Exception as e:
print(e)
add to AtlasI2C init statement about csv file name:
self.csv_file = <my_filename>.csv # replace my_filename with ur name
and then under line 51 (char_list = list(map(lambda x: chr(ord(x) & ~0x80), list(response[1:]))) add this line:
self.update_file(''.join(char_list))
Hope its gonna help You.
Cheers,
Fenrir
I have a scraping code within a for loop, but it would take several hours to complete, and the program stops when my Internet connection breaks. What I (think I) need is a condition at the beginning of the scraper that tells Python to keep trying at that point.
I tried to use the answer from here:
for w in wordlist:
#some text processing, works fine, returns 'textresult'
if textresult == '___': #if there's nothing in the offline resources
bufferlist = list()
str1=str()
mlist=list() # I use these in scraping
br = mechanize.Browser()
tried=0
while True:
try:
br.open("http://the_site_to_scrape/")
# scraping, with several ifs. Each 'for w' iteration results with scrape_result string.
except (mechanize.HTTPError, mechanize.URLError) as e:
tried += 1
if isinstance(e,mechanize.HTTPError):
print e.code
else:
print e.reason.args
if tried > 4:
exit()
time.sleep(120)
continue
break
Works while I'm online. When the connection breaks, Python writes the 403 code and skips that word from wordlist, moves on to the next and does the same. How can I tell Python to wait for connection within the iteration?
EDIT: I would appreciate it if you could write at least some of the necessary commands and tell me where they should be placed in my code, because I've never dealt with exception loops.
EDIT - SOLUTION I applied Abhishek Jebaraj's modified solution. I just added a very simple exception handling command:
except:
print "connection interrupted"
time.sleep(30)
Also, Jebaraj's getcode command will raise an error. Before r.getcode, I used this:
import urllib
r = urllib.urlopen("http: the site ")
The top answer to this question helped me as well.
Write another while loop inside which will keep trying to connect to the internet.
It will break only when it receives status code of 200 and then you can continue with your program.
Kind of like
retry = True
while retry:
try:
r = br.open(//your site)
if r.getcode()/10==20:
retry = False
except:
// code to handle any exception
// rest of your code
I have created a chat bot for Twitch IRC, I can connect and create commands etc etc, however I cannot use keyboard-interrupt in the command prompt. I suspect it is because it's stuck in this infinite loop, and I don't know how to fix this? I am new to programming btw!
Here is the code I have in my Run.py, openSocket() is defined in another file, basically connection to the server. s = socket.socket.
First part in the while-loop basically just reads the server messages, I think it's pretty straight forward for you guys!
s = openSocket()
joinRoom(s)
readbuffer = ""
while True:
readbuffer = readbuffer + s.recv(1024).decode("utf-8")
temp = str.split(readbuffer, "\n")
readbuffer = temp.pop()
for line in temp:
if "PING" in line:
s.send("PONG :tmi.twitch.tv\r\n".encode("utf-8"))
print("---SENT PONG---")
printMessage(getUser, getMessage, line)
message = getMessage(line)
for key in commands:
command = key
if command in message:
sendMessage(s, commands[command])
((Edit: I also have this problem where the connection to the server seems to time out for whatever reason. I managed to get it keep connection with ping/pong for about 40-45min, but then it disconnected again.
EDIT:
Sorry the original post was super messy. I have created this pastebin with the least amount of code I could use to recreate the problem.
If the IRC chat is inactive it will disconnect, and I can't get it to send 2 pings in a row without any messages in between, not sure if that's because it disconnects before the 2nd ping or because of the 2nd ping.
On at least one occasion it has disconnected even before I got the first ping from the server.
Pastebin: pastebin.com/sXUW50sS
Part of code that you posted doesn't have much to do with problem you described.
This is a guess (although an educated one). In you socket connection you are probably using try: except: and using Pokemon approach (gotta catch 'em all)
Thing here would be to find a line where you are doing something like this:
except:
pass
and change it to:
except (KeyboardInterrupt, SystemExit):
raise
except:
pass
Obviously I'm not trying to say here that your porgram should catch all exceptions and just pass like if nothing happened. Main point is that you are probably already doing that (for i-have-no-idea-why reasons) and you should have special treatment for system errors.
I am using a socket connection to download data through a third party API. It works fine for a while but every now and then my script will crash giving the following error: BrokenPipeError: [Errno 32] Broken pipe
After some research it seems the suggestion (link here) is to do the following:
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
However im firstly not sure what this actually does (im still confused after reading the python manual on signal). And I also don't know where to put the code.
If anyone is familiar with this error please could you advise if this is infact the correct solution and where the signal(SIGPIPE,SIG_DFL) would be placed. Should there be a try/except block inside which this is placed, or is it simply placed at the start of the program? Im confused.
Here's some of the relevant code. I basically have a dataframe consisting of several thousand items. I loop through each item passing it to the download method. The download method downloads the data via the api and then writes it to a database. I then move to the next item to download.
def recv_data(sock, recv_buffer=4096, delim='\n'):
buffer = ''
data = True
while data:
data = sock.recv(recv_buffer)
buffer += str(data.decode('latin-1'))
while buffer.find(delim) != -1:
line, buffer = buffer.split('\n', 1)
yield line
def update_existing_symbol_data(engine, sock, exchange, exchange_id, symbol, symbol_id, start_date):
data = ''
message = #request data message
sock.sendall(message.encode())
for line in recv_data(sock):
if "!ENDMSG!" in line:
break
data += line[:-2] + '\n'
df = pd.read_csv(io.StringIO(data))
df.set_index('date', inplace=True)
df.to_sql('daily', engine, if_exists='append')
def main():
df = #dataframe all symbols that need to be downloaded
for index, row in df.iterrows():
update_existing_symbol_data(args)
SIGPIPE is a POSIX thing that gets sent when a socket write operation fails. The default behavior is for the signal (this is an OS/socket thing, not a Python thing) to just kill your process. Python instead gives it to you as an exception so that it's possible to write more robust programs. But if you don't need to handle that event, which it sounds like you don't considering your use case, you can safely ignore it. There's no logic you need to do when you receive the signal, so the solution from that blog post should be fine. No try/except needed.
If your use case changes at a later date and you do need to handle the SIGPIPE, then wrapping that in a try/except and handling it there would be the way to go.