The context of this question is that I am trying to write a program for assist in analysis of data. It should be written in python3, however the kind of data it is for is usually stored in a format that python cannot read.
There is a package to the read these data but it is only compatible with python2. In order to the read the data I therefore wanted to write a python2 scripts that reads the file and converts it into a numpy array. This I want to read in my python3 program. (The package in question is axographio1).
In generality what I want is this:
Given a (python2) script like
#reading.py
import numpy
x = numpy.random.random(size=10000)
run a (python3) that can somehow get x
#analyze.py
import matplotlib.pyplot as plt
#fancyfunction that gets x from reading.py
plt.plot(x)
plt.show()
It is important here that reading.py be executed by the python2 interpreter since it will not work with python3.
Have you tried to pickle the data
In python 2
import pickle
pickle.dumps(x)
In python 3
import pickle
pickle.load(x)
If I remember correctly, a better approach is to save your numpy array in a json file (maybe using panda over numpy) in python 2, and do the reverse in python 3
Something like that :
df = pandas.Data[...]
See http://www.thegeeklegacy.com/t/how-to-import-without-using-import-in-python/35/#post-103 for the details
Below is the solution I used.
From the subprocess module I used the function Popen with argument shell=True to call my python2 scripts from the python3 and collected the stdout. (This means everything has be printed to the console.)
This is python2 code I want to call in python3:
#readying.py
import axographio #this is the python2 package
import os #specific to the problem of reading data
import sys #needed to use arguments from the command line
if __name__=='__main__': #when y
path = sys.argv[1]
filename = sys.argv[2]
os.chdir(path)
file = axographio.read(filename)
# print 'done reading files'
data = file.data
no_measurements = len(data)
measurement_len = len(data[0])
print(no_measurements)
print(measurement_len)
for i, name in enumerate(file.names):
print(name)
for i in range(no_measurements):
for j in range(measurement_len):
print(data[i][j])
print('next measurement')
#this acts as a seperator when reading the data in python3, anything that cannot be converted to a float works
What this code does is simply take argument from the commad line. (Using the sys module they can be passed to a .py scripts in the command line in the form python2 script.py arg1 arg2 arg2.
The output of reading.py is given out using print statements. Note that I print every datum individually in order to avoid truncation of the output. In this way the output is the standard output stdout of the python2 call.
The python3 code is
#analyze.py
import subprocess
import numpy as np
def read_data(path, filename):
module_name = 'reading.py'
cmd = 'python2 '+module_name+' '+path+' '+filename
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
out, err = p.communicate()
#out is a bytes object
result = out.split(b'\n') #the output from different print statements is seperated by "b'\n'" in the bytes object
#everything below here is concerned with unpacking the output and structuring it as I needed it
output = []
i = 0
while i < len(result):
output.append(result[i].decode("utf-8"))
i+=1
ind = 0 #this should keep track of where in the output list we are
no_measurements = int(output[ind])
ind += 1
measurement_len = int(output[ind])
ind += 1
names = []
for i in np.arange(ind,ind+no_measurements):
names.append(output[i])
ind += 1
data = []
measurement = []
while ind<len(output):
try:
measurement.append(float(output[ind]))
ind+=1
except:
data.append(measurement)
measurement = []
ind+=1
data = np.array(data)
return names, data
What this does is to use the subprocess module to execute as a shell command python2 reading.py path filename. As mentioned above the stdout of this call is the output of its print statements. These are bytes objects encoded with 'UTF-8'. Using the decode method they can be converted to string objects which can then be changed in type using functions like float. (This is another reason why it is useful to print everything seperately since it is rather annoying to have to scan through a string to find arrays or numbers.) In readying.py data are seperated explicitely while being printed which makes it easy to structure them once they are read.
Tags: axographio in python3--read axograph files python3
Related
I'm using a script from a third party I can't modify or show (let's call it original.py) which takes a file and produces some calculations. At the end it ouputs a result (using the print statment).
Since I have many files I decided to make a second script that gets all wanted files and runs them through the original.py
1st get list of all files to run
2nd run each file through the original.py
3rd obtain results from each file
I have the 1st and 2nd step. However, the end result only saves the calculations from the last file it read.
import sys
import original
import glob
import os
fn=str(sys.argv[1])
for filename in sys.argv[1:]:
print(filename)
ficheiros = [f for f in glob.glob(fn)]
for ficheiro in ficheiros:
original.file = bytes(ficheiro,'utf-8')
original.function()
To summarize:
Knowing I can't change the original script (which is made with a print statement) how can I obtain the results for each loop? Is there a better way than using a for loop?.
The first script can be invoked with python original.py
It requires the file to be changed manually inside the script in the original.file line.
This script outputs the result in the console and I redirect it with: python original.py > result.txt
At the moment when I try to run my script, it reads all the correct files in the folder but only returns the results for the last file.
#
(I tried to reformulate the question hopefully it's easier to understand)
#
The problem is due to a mistake in the ````ficheiros = [f for f in glob.glob(fn)]`````it's only reading one file, hence only outputting one result.
Thanks for the time.sleep() trick in the comments.
Solved:
I changed the initial part to:
fn=str(sys.argv[1])
ficheiros= []
for filename in sys.argv[1:]:
ficheiros.append(filename)
#print(filename)
and now it correctly reads all the files and it outputs all the results
Depending on your operating system there are different ways to take what is printed to the console and append it to a file.
For example on Linux, you could run this file that calls original.py for every file python yourfile.py >> outputfile.txt, which will then effectively save everything that is printed into outputfile.txt.
The syntax is similar for Windows.
I'm not quite sure what you're asking, but you could try one of these:
Either redirecting all output to a file for later use, by running the script like so: python secondscript.py > outfilename.txt
Or, and this might or might not work for you, redefining the print command to a function that outputs the result how you want, eg:
def print(x):
with open('outfile.txt','w') as f:
f.write('example: ' + x)
If you choose the second option, I recommend saving the old print function (oldprint = print) so you can restore and use the regular print later.
I don't know if I got exactly what you want. You have a first script named original.py which takes some arguments and returns things in the form of print statements and you would like to grab these prints statements in your scripts to do things?
If so, a solution could be the subprocess module:
Let's say that this is original.py:
print("Hi, I'm original.py")
print("print me!")
And this is main.py:
import subprocess
script_path = "original.py"
print("Executing ", script_path)
process = subprocess.Popen(["python3", script_path], stdout=subprocess.PIPE)
for line in process.stdout:
print(line.decode("utf8"))
You can easily add more arguments in the Popen call like ["arg1", "arg2",] etc.
Output:
Executing original.py
Hi, I'm original.py
print me!
and you can grab the lines in the main.py to do what you want with them.
I try do do the following:
I want to use the code described in this answer (Solution 1) to pass arrays of data between excel-VBA and Python. Reason for this is that I cannot find a scipy method for VBA and therefore I do it like this. To catch error messages from python I tried to implement something like this.
My VBA code looks like this:
Sub Usage2()
Dim outputarr() As Double
Dim oShell As Object, oCmd As String
Dim oExec As Object, oOutput As Object
Dim arg As Variant
Dim s As String, sLine As String
Set oShell = CreateObject("WScript.Shell")
oCmd = "<fullpathtopython> " & """" & "<fullpathtoscript>" & """"
' Make the data available via GetData()'
Cache = Array(4, 6, 8, 9)
Set oExec = oShell.Exec(oCmd)
Set oOutput = oExec.StdOut
While Not oOutput.AtEndOfStream
sLine = oOutput.ReadLine
If sLine <> "" Then s = s & sLine & vbNewLine
Wend
Debug.Print s
' Handle the returned data '
Debug.Assert Cache(3) = 8
Set oOutput = Nothing: Set oExec = Nothing
Set oShell = Nothing
End Sub
my python script looks like this:
import win32com.client
import numpy as np
import os
import matplotlib.pyplot as plt
import win32com.client
from scipy.interpolate import LSQUnivariateSpline, UnivariateSpline
print "Hello, User!"
# get the running instance of Excel
app = win32com.client.GetObject(Class="Excel.Application")
# get some data from Excel
data = app.run("GetData") # this OR
ssc=np.array(data) # this probably makes an error
print ssc
#do some fitting ...
# return some data to excel
app.run("SetData", sscnew)
The idea is simple:
call python script using shell.run or shell.exec(oCmd)
python script gets it's input from VBA
python does stuff
python passes data back to VBA
print error messages that occured
The problem with this VBA code is, that Python.exe gets opened but the program does not get executed. It looks like this:
Open shell and blinking cursor. The 'Hello, User' doesn't get printed so the program doesn't get executed since this usually runs through before any errors occure in the program. If I close this window, 'Hello, User!' gets printed in VBA but nothing else.
Specific question: How does one input and output arrays via this VBA-Python interface AND get the errormessages to print them in the VBA-shell?
One option is processing "stuff' with pandas and save as csv, then pull into VBA as array(s) or even do everything in python before pulling the end result in VBA for formatting, emailing, whatever you need VBA for.
regarding the error, had almost the same issue and was able to resolve thank to #MichaelButscher . link. Basically you need to use error redirecting as in
import sys
path_err = r'\\yourpath\test\err.txt'
path_out = r'\\yourpath\test\out.txt'
sys.stderr = open(path_err, 'w')
sys.stdout = open(path_out, 'w')
path = r'\\yourpath\test\test1.txt'
with open (path, "w") as f:
f.write("test1")
raise IndexError
to track fatal errors (err.txt) or expected output (out.txt).
I hope this helps
It's propably very basic question but I couldn't find any answer. Right now I have something like:
import sys
inFile = sys.argv[1]
outFile = sys.argv[2]
with open(inFile, 'r+') as input,open(outFile,'w+') as out:
#dosomething
I can run it with
./modulname foo bar (working). How can I change it so it will work with /.modulname foo > bar? (right now it gives me following error).
./pagereport.py today.log > sample.txt
Traceback (most recent call last):
File "./pagereport.py", line 7, in <module>
outFile = sys.argv[2]
IndexError: list index out of range
You could skip the second open (out) and instead use sys.stdout to write to.
If you want to be able to use both ways of calling it, argparse has a comfortable way of doing that with add_argument by combining type= to a file open for writing and making sys.stdout its default.
When you do:
./modulname foo > bar
> is acted upon by shell, and duplicates the STDOUT stream (FD 1) to the file bar. This happens before the command even runs, so no, you can't pass the command like that and have bar available inside the Python script.
If you insist on using >, a poor man's solution would be to make the arguments a single string, and do some string processing inside, something like:
./modulname 'foo >bar'
And inside your script:
infile, outfile = map(lambda x: x.strip(), sys.argv[1].split('>'))
Assuming no filename have whitespaces, take special treatment like passing two arguments in that case.
Also, take a look at the argparse module for more flexible argument parsing capabilities.
What error have you got?
import sys
inFile = sys.argv[1]
outFile = sys.argv[2]
with open(inFile, 'r+') as in_put ,open(outFile,'w+') as out:
buff = in_put.read()
out.write(buff)
I try to run you code, but you have no import sys , so after fixed it as above . I can run it as a simple cp command.
python p4.py p4.py p4.py-bk
I am using python and I am supposed to read a file from command line for further processing. My input file has a binary that should be read for further processing. Here is my input file sub.py:
CODE = " \x55\x48\x8b\x05\xb8\x13\x00\x00"
and my main file which should read this is like the following:
import pyvex
import archinfo
import fileinput
import sys
filename = sys.argv[-1]
f = open(sys.argv[-1],"r")
CODE = f.read()
f.close()
print CODE
#CODE = b"\x55\x48\x8b\x05\xb8\x13\x00\x00"
# translate an AMD64 basic block (of nops) at 0x400400 into VEX
irsb = pyvex.IRSB(CODE, 0x1000, archinfo.ArchAMD64())
# pretty-print the basic block
irsb.pp()
# this is the IR Expression of the jump target of the unconditional exit at the end of the basic block
print irsb.next
# this is the type of the unconditional exit (i.e., a call, ret, syscall, etc)
print irsb.jumpkind
# you can also pretty-print it
irsb.next.pp()
# iterate through each statement and print all the statements
for stmt in irsb.statements:
stmt.pp()
# pretty-print the IR expression representing the data, and the *type* of that IR expression written by every store statement
import pyvex
for stmt in irsb.statements:
if isinstance(stmt, pyvex.IRStmt.Store):
print "Data:",
stmt.data.pp()
print ""
print "Type:",
print stmt.data.result_type
print ""
# pretty-print the condition and jump target of every conditional exit from the basic block
for stmt in irsb.statements:
if isinstance(stmt, pyvex.IRStmt.Exit):
print "Condition:",
stmt.guard.pp()
print ""
print "Target:",
stmt.dst.pp()
print ""
# these are the types of every temp in the IRSB
print irsb.tyenv.types
# here is one way to get the type of temp 0
print irsb.tyenv.types[0]
The problem is that when I run "python maincode.py sub.py' it reads the code as content of the file but its output is completely different from when I directly add CODE into the statement irsb = pyvex.IRSB(CODE, 0x1000, archinfo.ArchAMD64()). Does anyone know what is the problem and how can I solve it? I even use importing from inputfile but it does not read a text.
Have you considered the __import__ way?
You could do
mod = __import__(sys.argv[-1])
print mod.CODE
and just pass the filename without the .py extension as your command line argument:
python maincode.py sub
EDIT: Apparently using __import__ is discouraged. Instead though you can use importlib module:
import sys,importlib
mod = importlib.import_module(sys.argv[-1])
print mod.CODE
..and it should work the same as using __import__
If you need to pass a path to the module, one way is if in each of the directories you added an empty file named
__init__.py
That will allow python to interpret the directories as module namespaces, and you can then pass the path in its module form: python maincode.py path.to.subfolder.sub
If for some reason you cannot or don't want to add the directories as namespaces, and don't want to add the init.py files everywhere, you could also use imp.find_module. Your maincode.py would instead look like this:
import sys, imp
mod = imp.find_module("sub","/path/to/subfolder/")
print mod.code
You'll have to write code which breaks apart your command line input into the module part "sub" and the folder path "/path/to/subfolder/" though. Does that make sense? Once its ready you'll call it like you expect, python maincode.py /path/to/subfolder/sub/
you're reading the code as text, while when reading as file you're likely reading as binary
you probably need to convert binary to text of vice-versa to make this work
Binary to String/Text in Python
I want to read all files in a directory and pass them via command line to another program.
The following is a part of my code (for one file here) which does not seem to work, and I don't really understand why it would not work.
My code (with a bit of debug print):
# -*- coding: iso-8859-15 -*-
# Python 3
avidemux_dir = "C:\\Program Files (x86)\\Avi Demux\\avidemux.exe"
start_dir = "F:\\aaa" # without ending backslash!
extension = ".mpg"
import os
import subprocess
for dirpath, dirnames, filenames in os.walk(start_dir):
if filenames:
first_file = os.path.join(dirpath, filenames[0])
test2 = "--load " + first_file
print(dirpath) #results in: F:\aaa\av01
print(first_file) #results in: F:\aaa\av01\av01.mpg
print(test2) #results in: --load F:\aaa\av01\av01.mpg
p1 = subprocess.Popen([avidemux_dir, "--load", first_file])
p2 = subprocess.Popen([avidemux_dir, test2])
For this example, avidemux will work (load the correct file) only for p1. p2 does not work.
Why is that?
The commandline example that works in .bat:
avidemux.exe --load F:\aaa\av01\av01.mpg
I really would like to have it all in one string like in p2 because I join a larger list of files together to one big string with the correct variables for avidemux.
shlex is one approach, but from the files paths it's obvious you're running on Windows, and shlex assumes conventions used in Unix-ish shells. They can get you in trouble on Windows.
As the docs say, the underlying Windows API call takes a single string as an argument, so on Windows you're generally much better off passing a single string to Popen().
Oops! I see you've already discovered that. But at least now you know why ;-)
use
import shlex
p2 = subprocess.Popen([avidemux_dir] + shlex.split(test2))
see the docs about command args of Popen.
Ah, you're passing a string of two arguments there. You need to split it, if necessary using shlex.split:
p2 = subprocess.Popen([avidemux_dir, *shlex.split(test2)])
Or just pass a string:
p2 = subprocess.Popen(avidemux_dir + ' ' + test2, shell=True)
Just stumbled across the solution: Not using a list, when doing something like that.
Solution:
test2 = avidemux_dir + " --load " + first_file
and
p2 = subprocess.Popen(test2) # no more list but the pure string.