On the latest version of Debian 32-bit with Python 2.7.3, I've compiled Plink (part of the PuTTY suite of tools) from source. For those unfamiliar, Plink is a great tool to issue commands on SSH servers so you can script your commands (I've found them wonderful for Cisco switches, which is what I'm doing here).
I have a file called switch.list containing names of switches on each line, such as:
Net-Switch-1
Net-Switch-2
Backbone-1
Now my Python script looks like this:
import subprocess
Switches = []
SwitchFile = open("switch.list")
for line in SwitchFile:
Switches.append(line)
SwitchFile.close()
for sw in Switches:
p = subprocess.Popen(["./plink","-ssh","-l","admin","-pw","REDACTED","-noagent","-batch",sw,"show","clock"], stdout=subprocess.PIPE)
print p.communicate()
My output is:
Unable to open connection:
Name or service not known
('', None)
Over and over, for as many times as my switch count. That tells me it's reading the file and populating the array just fine, but the Plink for-loop is messed up.
Troubleshooting: If I replace sw with a hard-coded switch name, like Net-Switch-1, then it runs fine. This is why I know the variable, sw, isn't being passed along correctly.
More Troubleshooting: If I run the Plink command from CLI, omitting the switch name, I get the same error output, but without the third line of "('', None)"
Troubleshooting where I start to get tricky: This doesn't work either:
p = subprocess.Popen(["./plink","-ssh","-l","admin","-pw","REDACTED","-noagent","-batch",(" "+sw),"show","clock"], stdout=subprocess.PIPE)
When reading from file, lines contain the terminating new-line character. Try this instead:
for line in SwitchFile:
Switches.append(line.strip())
Note also, that the common practice is to call variables with lower-case names.
Are you reading the file correctly? If the info in the file is separated by newlines, Try something different like.
with open(file_path, 'r') as f:
data = f.readlines() #extracts to a list.
print data
Related
Having an odd problem.
I have a Django app that opens a file (represented as a Django FieldFile) and reads each row using readline() as below:
with file.open(mode='r') as f:
row = f.readline()
# do something with row...
The file is text, utf-8 encoded and lines are terminated with \r\n.
The problem is each row is being read as the hex representation of the string, so instead of "Hello" I get "48656c6c6f".
A few stranger things:
It previously worked properly, but at some point an update has broken it (I've tried rolling back to previous commits and it is still wonky, so possibly a dependency has updated and not something from my requirements.txt). Missed it in my testing because it is in a very rarely used part of the app.
If I read the same file using readlines() instead of readline() I see the correct string representation of the file wrapped in [b'...']
The file reads normally if I do it using straight Python open() and readline() from an interpreter
Forcing text mode with mode='rt' doesn't change the behaviour, neither does mode='rb'
The file is stored in a Minio bucket, so the defaut storage is storages.backends.s3boto3.S3Boto3Storage from django-storages and not the default Django storage class. This means that boto3, botocore and s3fs are also in the mix, making it more confusing for me to debug.
Scratching my head at why this worked before and what I'm doing wrong.
Environment is Python 3.8, Django 2.2.8 and 3.0 (same result) running in Docker containers.
EDIT
Let me point out that the fix for this is simply using
row = f.readline().decode()
but I would still like to figure out what's happening.
EDIT 2
Further to this, FieldFile.open() is reading the file as a binary file, whereas plain Python open() is reading the file as a text file.
This seems very weird.
I think you will see the solution immediately after trying following (I will then update my answer or delete it if it really doesn't help to find it, but I'm quite confident)
A assume, that there is some code, that is monkeypatching file.open or the django view function.
What I suggest is:
Start your code with manage.py runserver
Ad following code to manage.py (as the very first lines)
import file
print("ID of file.open at manage startup is", id(file.open)
Then add code to your view directly one line above the file.open
print("ID of file.open before opening is", id(file.open)
If both ids are different, then something monkeypatched your open function.
If both are the same, then the problem must be somewhere else.
If you don not see the output of these two prints, something might have monkeypatched your view.
If this doesn't work, then try to use
open() instead of file.open()
Is there any particular reason you use file.open()
Addendum 1:
So what you sai is, that file is an object instance of a class is it a FileField?
In any case can you obtain the name of the file and open it with a normal open() to see whether it is only file.open() that does funny things or whether it is also open() reading it this stange way.
Did you just open the file from command line with cat filename (or if under windows with type filename?
If that doesn't work we could add traces to follow each line of the source code that is being executed.
Addendum 2:
Well if you can't try this in a manage.py runserver, what happens if you try to read the file with a manage.py shell?
Just open the shell and type something like:
from <your_application>.models import <YourModel>
entry = <YourModel>.objects.get(id=<idofentry>)
line1 = entry.<filefieldname>.open("r").read().split("\n")[0]
print("line1 = %r" % line1)
If this is still not conclusive, (but only if you can reproduce the issue with the management shell, then create a small file containing the lines.
from <your_application>.models import <YourModel>
entry = <YourModel>.objects.get(id=<idofentry>)
import pdb; pdb.set_trace()
line1 = entry.<filefieldname>.open("r").read().split("\n")[0]
print("line1 = %r" % line1)
And import it from the management shell.
The code should enter the debugger and now you can single step through the open function and see whether you end up on sime weird function in some monkeypatch.
I'm running a binary that manages a usb device. The binary file, when executed outputs results to a file I specify.
Is there any way in python the redirect the output of a binary to my script instead of to a file? I'm just going to have to open the file and get it as soon as this line of code runs.
def rn_to_file(comport=3, filename='test.bin', amount=128):
os.system('capture.exe {0} {1} {2}'.format(comport, filename, amount))
it doesn't work with subprocess either
from subprocess import check_output as qx
>>> cmd = r'C:\repos\capture.exe 3 text.txt 128'
>>> output = qx(cmd)
Opening serial port \\.\COM3...OK
Closing serial port...OK
>>> output
b'TrueRNG Serial Port Capture Tool v1.2\r\n\r\nCapturing 128 bytes of data...Done'
The actual content of the file is a series of 0 and 1. This isn't redirecting the output to the file to me, instead it just prints out what would be printed out anyway as output.
It looks like you're using Windows, which has a special reserved filename CON which means to use the console (the analog on *nix would be /dev/stdout).
So try this:
subprocess.check_output(r'C:\repos\capture.exe 3 CON 128')
You might need to use shell=True in there, but I suspect you don't.
The idea is to make the program write to the virtual file CON which is actually stdout, then have Python capture that.
An alternative would be CreateNamedPipe(), which will let you create your own filename and read from it, without having an actual file on disk. For more on that, see: createNamedPipe in python
I'm new to scripting and have been reading up on Python for about 6 weeks. The below is meant to read a log file and send an alert if one of the keywords defined in srchstring is found. It works as expected and doesn't alert on strings previously found, as expected. However the file its processing is actively being written to by an application and the script is too slow on files around 500mb. under 200mb it works fine ie within 20secs.
Could someone suggest a more efficient way to search for a string within a file based on a pre-defined list?
import os
srchstring = ["Shutdown", "Disconnecting", "Stopping Event Thread"]
if os.path.isfile(r"\\server\\share\\logfile.txt"):
with open(r"\\server\\share\\logfile.txt","r") as F:
for line in F:
for st in srchstring:
if st in line:
print line,
#do some slicing of string to get dd/mm/yy hh:mm:ss:ms
# then create a marker file called file_dd/mm/yy hh:mm:ss:ms
if os.path.isfile("file_dd/mm/yy hh:mm:ss:ms"): # check if a file already exists named file_dd/mm/yy hh:mm:ss:ms
print "string previously found- ignoring, continuing search" # marker file exists
else:
open("file_dd/mm/yy hh:mm:ss:ms", 'a') # create file_dd/mm/yy hh:mm:ss:ms
print "error string found--creating marker file sending email alert" # no marker file, create it then send email
else:
print "file not exist"
Refactoring the search expression to a precompiled regular expression avoids the (explicit) innermost loop.
import os, re
regex = re.compile(r'Shutdown|Disconnecting|Stopping Event Thread')
if os.path.isfile(r"\\server\\share\\logfile.txt"):
#Indentation fixed as per comment
with open(r"\\server\\share\\logfile.txt","r") as F:
for line in F:
if regex.search(line):
# ...
I assume here that you use Linux. If you don't, install MinGW on Windows and the solution below will become suitable too.
Just leave the hard part to the most efficient tools available. Filter your data before you go to the python script. Use grep command to get the lines containing "Shutdown", "Disconnecting" or "Stopping Event Thread"
grep 'Shutdown\|Disconnecting\|"Stopping Event Thread"' /server/share/logfile.txt
and redirect the lines to your script
grep 'Shutdown\|Disconnecting\|"Stopping Event Thread"' /server/share/logfile.txt | python log.py
Edit: Windows solution. You can create a .bat file to make it executable.
findstr /c:"Shutdown" /c:"Disconnecting" /c:"Stopping Event Thread" \server\share\logfile.txt | python log.py
In 'log.py', read from stdin. It's file-like object, so no difficulties here:
import sys
for line in sys.stdin:
print line,
# do some slicing of string to get dd/mm/yy hh:mm:ss:ms
# then create a marker file called file_dd/mm/yy hh:mm:ss:ms
# and so on
This solution will reduce the amount of work your script has to do. As Python isn't a fast language, it may speed up the task. I suspect it can be rewritten purely in bash and it will be even faster (20+ years of optimization of a C program is not the thing you compete with easily), but I don't know bash enough.
I am running a python script on a screen on linux server and when I do TOP command I can see it running but it has been for hours that the script did not write anything. Anybody know what could be the reason?
Here is my script:
import GeoIP
from netaddr import *
gi = GeoIP.GeoIP("/data/GeoIPOrg_20141202.dat", GeoIP.GEOIP_MEMORY_CACHE)
o = Path to the output text file
for line in f:
line = line.strip('\n')
asn,ip,count =line.split('|')
org = gi.org_by_addr(ip)
start,end = gi.range_by_ip(ip)
ran = list(IPRange(start,end))
# ipcount = len(ran)
ip_start,ip_end = IPAddress(start),IPAddress(end)
n_start = int(ip_start)
n_end = int(ip_end)
range = (n_start,n_end)
print ("%s|%s|%s|%s|%s|%s" % (asn,range,len(ran),org,ip,count) , file = o)
This could be a few things; hard to say without seeing how you're running and how you're initialising that file.
A definite possibility is the file isn't being flushed (more relevantly, see the docs on changing the buffer size of open() as it's probably being invoked in your code).
Either way it's worth using Python (2.5+)'s with statement to handle file / resource management neatly and robustly instead of relying on print e.g.:
with open("/my/output/path.txt", "w") as out_file:
# Rest of code
# ...
out_file.write("%s|%s|%s|%s|%s|%s\n" % (asn,range,len(ran),org,ip,count))
See this SO question for good examples of using the with statement.
You have two ways to achieve this.
You can change your code to use the with statement (context
manager) as #Nick B has suggested in his answer to open your file
there.
Or you can set the buffering where you open the file to be
line buffering.
So where you say:
# Im assuming you open your file like this since your code is
# an incomplete snippet. Otherwise tell us how you open your file
o = open('output_file.log', 'w')
You must say:
o = open('output_file.log', 'w', buffering=1) # enable line buffering
You should read the help of then open command by typing help(open) in the interactive python shell. It explains a great deal how buffering works in python.
I am writing a python script that reads a line/string, calls Unix, uses grep to search a query file for lines that contain the string, and then prints the results.
from subprocess import call
for line in infilelines:
output = call(["grep", line, "path/to/query/file"])
print output
print line`
When I look at my results printed to the screen, I will get a list of matching strings from the query file, but I will also get "1" and "0" integers as output, and line is never printed to the screen. I expect to get the lines from the query file that match my string, followed by the string that I used in my search.
call returns the process return code.
If using Python 2.7, use check_output.
from subprocess import check_output
output = check_output(["grep", line, "path/to/query/file"])
If using anything before that, use communicate.
import subprocess
process = subprocess.Popen(["grep", line, "path/to/query/file"], stdout=subprocess.PIPE)
output = process.communicate()[0]
This will open a pipe for stdout that you can read with communicate. If you want stderr too, you need to add "stderr=subprocess.PIPE" too.
This will return the full output. If you want to parse it into separate lines, use split.
output.split('\n')
I believe Python takes care of line-ending conversions for you, but since you're using grep I'm going to assume you're on Unix where the line-ending is \n anyway.
http://docs.python.org/library/subprocess.html#subprocess.check_output
The following code works with Python >= 2.5:
from commands import getoutput
output = getoutput('grep %s path/to/query/file' % line)
output_list = output.splitlines()
Why would you want to execute a call to external grep when Python itself can do it? This is extra overhead and your code will then be dependent on grep being installed. This is how you do simple grep in Python with "in" operator.
query=open("/path/to/query/file").readlines()
query=[ i.rstrip() for i in query ]
f=open("file")
for line in f:
if "line" in query:
print line.rstrip()
f.close()