Develop Reference

Passing a temporary file created with Python's tempfile to a command line program ignoring line ending preferences

I've been banging my head against the wall at this for going on 16 hours and I can't seem to find anybody with the exact same issue.
I have a command line program (I'll just call it the CLI) I'm using that takes as an argument a text file, where each line of the file has important info. Without going into detail, the CLI passes the lines of the file to a web service where some operations are done.
I have a python program I'm writing where I create the file that gets passed to the CLI programmatically (using subprocess). The issue seems to be that the webservice is receiving a carriage return somewhere which is doesn't know how to handle. I was pretty careful to avoid any carriage returns in my creation of each of the lines that go into that file, but clearly carriage returns were getting added to the end. I opened my file in binary mode and sure enough, there was the carriage return. Bummer.
No matter, I see a script shared around here all the time showing how to convert a file from CRLF to LF. Added those handful of lines to my python script, and double checked and sure enough, when reading the file in binary mode, the carriage return was gone. hurray.
But when I go to run the script again, for some reason, the web service is still receiving carriage returns.
I suspect it has something to do with the python temporary file system I'm using. Maybe it works some weird way under the hood and after i'm changing the file to use LF and printing its contents, at some point its changing it back (maybe when its passed as an argument to the CLI)
I guess my question is, either 1) Can you think of something tempfile might be doing to mess with my file endings even after it looks like I had changed them? Maybe when the temp file gets passed into the cli with subprocess its reverting somehow? I'm just sending it with temp.name.
Or 2) Maybe I can do something other than python's built in tempfile. Its really convenient, so if I were going to not use it, are there some best practices for creating and deleting temporary files manually I should know about? Is there a best place to create them? For example, is it considered bad practice to create and then delete files right off the current directory?
Anyway, here's the relevant part of my code AFTER the changes to the line endings:
def my_function(mylist, my_id, local_id, args):
temp = tempfile.NamedTemporaryFile(mode='w+b', delete=False)
for each in mylist:
is_directory = False
destination = args.destination.replace("\\", "/")
if each["smallpath"] == "/":
full_path = each["smallpath"] + "/"
else:
full_path = each["rel_path"].replace("\\", "/") + "/" + each["specific_path"].lstrip("/")
if os.path.basename(full_path) == "":
is_directory = True
if is_directory is False:
line = f'"{full_path}" "~/{destination}/{each["orgID"]}-{each["userID"]}/{os.path.basename(full_path)}" \n'
else:
if each["smallpath"] != "/":
slash_index = full_path.rstrip('/').rfind("/")
local_dir = full_path[slash_index:].rstrip().rstrip('/')
else:
local_dir = "/"
line = f'"{full_path}" "~/{destination}/{each["orgID"]}-{each["userID"]}/{local_dir.lstrip("/")}" --recursive \n'
line = line.replace("\\", "/")
temp.write(line)
temp.seek(0)
windows_ending = b'\r'
unix_ending = b'\n'
double_newline = b'\n\n'
with open(temp.name, 'rb') as file:
file_contents = file.read()
temp.seek(0)
file_contents = file_contents.replace(windows_ending, unix_ending)
file_contents = file_contents.replace(double_newline, unix_ending)
with open(temp.name, 'wb') as file:
file.write(file_contents)
myprocess = subprocess.Popen(["programname", "commandname", myid, local_id, "--batch",
temp.name], stdout=subprocess.PIPE)
mycommand = myprocess.communicate()[0].decode('utf-8')
temp.close()
os.unlink(temp.name)

I wanted to thank everyone who tried to help. The comments helped me discover another vulnerability elsewhere in my code. The problem ended up actually not being my own, rather it was the service I was using

Python for loop not running

Im writing a simple command line program involving id's, i want to make it so that every time i run a function it should add a value to a CSV file. I have made a function that i thought worked, but apparently not. One of my for loops is not running correctly ;( Here is my code:
def addId(id):
file = open(ID_FILE_PATH, 'wb+')
read = csv.reader(file)
write = csv.writer(file)
existingRows = []
existingRows.append(id)
for rows in read: # This does not run
print rows
if len(rows) > 0 and rows[0] not in existingRows:
existingRows.append(rows[0])
for rows in existingRows: # This runs
write.writerow([rows])
file.close()
Sorry for my bad English btw.

You open the file with:
file = open(ID_FILE_PATH, 'wb+')
According to the documentation:
note that 'w+' truncates the file
You truncate the file, so no wonder there is nothing to read in! Use rb+ instead.

Python conditional statement based on text file string

Noob question here. I'm scheduling a cron job for a Python script for every 2 hours, but I want the script to stop running after 48 hours, which is not a feature of cron. To work around this, I'm recording the number of executions at the end of the script in a text file using a tally mark x and opening the text file at the beginning of the script to only run if the count is less than n.
However, my script seems to always run regardless of the conditions. Here's an example of what I've tried:
with open("curl-output.txt", "a+") as myfile:
data = myfile.read()
finalrun = "xxxxx"
if data != finalrun:
[CURL CODE]
with open("curl-output.txt", "a") as text_file:
text_file.write("x")
text_file.close()
I think I'm missing something simple here. Please advise if there is a better way of achieving this. Thanks in advance.

The problem with your original code is that you're opening the file in a+ mode, which seems to set the seek position to the end of the file (try print(data) right after you read the file). If you use r instead, it works. (I'm not sure that's how it's supposed to be. This answer states it should write at the end, but read from the beginning. The documentation isn't terribly clear).
Some suggestions: Instead of comparing against the "xxxxx" string, you could just check the length of the data (if len(data) < 5). Or alternatively, as was suggested, use pickle to store a number, which might look like this:
import pickle
try:
with open("curl-output.txt", "rb") as myfile:
num = pickle.load(myfile)
except FileNotFoundError:
num = 0
if num < 5:
do_curl_stuff()
num += 1
with open("curl-output.txt", "wb") as myfile:
pickle.dump(num, myfile)
Two more things concerning your original code: You're making the first with block bigger than it needs to be. Once you've read the string into data, you don't need the file object anymore, so you can remove one level of indentation from everything except data = myfile.read().
Also, you don't need to close text_file manually. with will do that for you (that's the point).

Sounds more for a job scheduling with at command?
See http://www.ibm.com/developerworks/library/l-job-scheduling/ for different job scheduling mechanisms.

The first bug that is immediately obvious to me is that you are appending to the file even if data == finalrun. So when data == finalrun, you don't run curl but you do append another 'x' to the file. On the next run, data will be not equal to finalrun again so it will continue to execute the curl code.
The solution is of course to nest the code that appends to the file under the if statement.

Well there probably is an end of line jump \n character which makes that your file will contain something like xx\n and not simply xx. Probably this is why your condition does not work :)
EDIT
What happens if through the python command line you type
open('filename.txt', 'r').read() # where filename is the name of your file
you will be able to see whether there is an \n or not

Try using this condition along with if clause instead.
if data.count('x')==24
data string may contain extraneous data line new line characters. Check repr(data) to see if it actually a 24 x's.

Check if a file is modified in Python

I am trying to create a box that tells me if a file text is modified or not, if it is modified it prints out the new text inside of it. This should be in an infinite loop (the bot sleeps until the text file is modified).
I have tried this code but it doesn't work.
while True:
tfile1 = open("most_recent_follower.txt", "r")
SMRF1 = tfile1.readline()
if tfile1.readline() == SMRF1:
print(tfile1.readline())
But this is totally not working... I am new to Python, can anyone help me?

def read_file():
with open("most_recent_follower.txt", "r") as f:
SMRF1 = f.readlines()
return SMRF1
initial = read_file()
while True:
current = read_file()
if initial != current:
for line in current:
if line not in initial:
print(line)
initial = current
Read the file in once, to get it's initial state. Then continuously repeat reading of the file. When it changes, print out its contents.
I don't know what bot you are referring to, but this code, and yours, will continuously read the file. It never seems to exit.

I might suggest copying the file to a safe duplicate location, and possibly using a diff program to determine if the current file is different from the original copy, and print the added lines. If you just want lines appended you might try to utilize a utility like tail
You can also use a library like pyinotify to only trigger when the filesystem detects the file has been modified

This is the first result on Google for "check if a file is modified in python" so I'm gonna add an extra solution here.
If you're curious if a file is modified in the sense that its contents have changed, OR it was touched, then you can use os.stat:
import os
get_time = lambda f: os.stat(f).st_ctime
fn = 'file.name'
prev_time = get_time(fn)
while True:
t = get_time(fn)
if t != prev_time:
do_stuff()
prev_time = t

How to append a list to a file on Python using JSON

I'm getting a bit of a trouble here. I have a text file with ["Data1", "Data2", "Data3"], and I want to make that if data1 is not in the file, then append a new list with all three strings, and if data is already there, then just print it. What is broken in this code and why?
filename = "datosdeusuario.txt"
leyendo = open(filename, 'r')
if user.name in leyendo:
Print("Your user name is already there")
else:
file = open(filename, 'a')
file.write(json.dumps([user.name, "data2", "data3"])+"\n")
file.close()
Print("Since I couldn't find it, I did append your name and data.")
P.S.: I am a rookie in Python, and I'm getting confused often. That's why I am not using any dicts (no idea what they are anyway), so I'd like to make that code work in the most simple way.
P.S.2: Also, if that works, my next step would be to make a search engine to return one specific of the three data items in the list. For example, if I want to get the data2 in a list with username "sael", what would I need to do?

It seems that you're reading from the file pointer, NOT from the data in the file as you expected.
So, you first need to read the data in the file:
buffer = leyendo.read()
Then do your check based on buffer, not leyendo:
if user.name in buffer:
Also, you're opening the file two times, that may be kind of expensive. I am not sure if Python got a feature to open the file in both read and write modes.

Assuming that your user.name and your Print functions are working, you need to read the file and close the file.
Try this:
filename = "datosdeusuario.txt"
f = open(filename, 'r')
leyendo = f.read()
f.close()
if user.name in leyendo:
Print("Your user name is already there")
else:
file = open(filename, 'a')
file.write(json.dumps([user.name, "data2", "data3"])+"\n")
file.close()
Print("Since I couldn't find it, I did append your name and data.")

First, you should close the file in both cases, and I think you should close the file before re-opening it for appending.
I think the problem is with the line:
if user.name in leyendo:
which will always return false.
You should read the file and then question it like so:
if user.name in leyendo.read():