Develop Reference

Why do I get a MemoryError in 64bit python when doing a replace within a 8MB file? (Bytes)

I have a mitmproxy flow that contains multiple image files. The flow itself is in bytes. I am trying to dump the old images to files, then replace them within the flow with my own images.
However, I hit a Memory Error in Python and I don't understand why. The flow file itself is 8mb, the pictures I am trying to replace the old ones with are small, less then 100kb. There should be plenty of memory for that?
startList = list(re.finditer(b'\xff\xd8',flowContent))
x = 1
for a in startList:
end = flowContent.find(b'\xff\xd9',a.start())
fileContent = flowContent[a.start():end]
fileName = 'image'+str(x)+".jpg"
dumpfile = open('dump/'+fileName,'wb')
dumpfile.write(fileContent)
dumpfile.close()
replace = open('replace/replace'+str(x)+'.jpg','rb')
myImage = Image(replace)
replace.close()
nowTime = datetime.now()
myImage.datetime = nowTime.strftime(DATETIME_STR_FORMAT)
myImage.datetime_digitized = nowTime.strftime(DATETIME_STR_FORMAT)
myImage.datetime_original = nowTime.strftime(DATETIME_STR_FORMAT)
newImage = open('replace/replace'+str(x)+'U.jpg',"wb")
newImage.write(myImage.get_file())
newImage.close()
replaceF = open('replace/replace'+str(x)+'U.jpg','rb')
replaceContent = replaceF.read()
replaceF.close()
flowContent = flowContent.replace(fileContent,replaceContent)
#flowContent = re.sub(fileContent,myImage.get_file(),flowContent)
x = x+1
The error is on this line:
Traceback (most recent call last):
File "E:\SpecialK\flow.py", line 41, in <module>
flowContent = flowContent.replace(fileContent,replaceContent)
MemoryError

pandas error on linux, but working on windows

So, i ran my "ml" model on my local windows machine, everything runs smooth, it just takes 48 hour to fully run every process, naturally i ask the company more procesing power to cut times, they give me a linux simulation server to run my models, but for some reason pandas is giving me the next error:
Traceback (most recent call last):
File "/ANACONDATA/Prediccion_de_Fallas/03_Modelos_y_Scripts/testv14.py", line 894, in <module>
dlist[xx][namer2] = np.where((dlist[xx].too_soon == 0),dlist[xx][column].shift(24) , 0)
File "/opt/anaconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3643, in __setitem__
self._setitem_array(key, value)
File "/opt/anaconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3702, in _setitem_array
self._iset_not_inplace(key, value)
File "/opt/anaconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3721, in _iset_not_inplace
raise ValueError("Columns must be same length as key")
this is the code where i fails (runs ok on windows), tried using pandas 1.3.5, and 1.4.2 same result
features=['AN_Utility_Supply_Press','AN_LPC_ASV_Position',
'AN_Eng_Brg_3Y_Gap',... 200 something list of features]
dlist = {}
turbo= np.unique(dfx2['SAP'])
for xx in (turbo):
dlist[xx]=dfx2.loc[(dfx2['SAP'] == xx)]
for column in dlist[xx][features]:
namer2=[column+'_'+'Lag']
fails here------>dlist[xx][namer2] = np.where((dlist[xx].too_soon == 0),dlist[xx][column].shift(24) , 0)
# namer3=[column+'_'+'Lchg'+"24"]
# dlist[xx][namer3] = np.where((dlist[xx].too_soon == 0),(dlist[xx][column]-dlist[xx][column].shift(24)) , 0)
namer4=[column+'_'+'mean']
dlist[xx][namer4] = np.where((dlist[xx].too_soon == 0),(dlist[xx][column].rolling(min_periods=1, window=feature_window).mean()), dlist[xx][column])
namer5=[column+'_'+'max']
dlist[xx][namer5] = np.where((dlist[xx].too_soon == 0),(dlist[xx][column].rolling(min_periods=1, window=feature_window).max()), dlist[xx][column])
dfx2 = pd.concat(dlist)
dfx2.reset_index(drop=True)
dfx2=dfx2.droplevel(level=0)
am i missing something?, why this happens?

Ok, took some time to figure it out, i tried more versions of pandas until it work, the version is 1.2.4 dont really have an explanation to what happen.

PyFMI multiple inputs in Master simulation

I am trying to simulate two FMUs with the one having inputs as CSV files by using the Master. What I have tried is the following:
from pyfmi import load_fmu
from pyfmi import Master
import pandas as pd
electricity_network = load_fmu(r"C:\Users\kosmy\Pandapower_Reduced.fmu")
pv = load_fmu(r"C:\Users\kosmy\Photovoltaics.Model.PVandWeather_simple.fmu")
load_file = r"C:\Users\kosmy\load_prof_sec_res.csv"
load = pd.read_csv(load_file)
models = [electricity_network, pv]
connections = [(pv, "P_MW", electricity_network, "P_pv1"),
(pv, "P_MW", electricity_network, "P_pv2")]
master_simulator = Master(models, connections)
input_object = [((electricity_network, 'P_load1'), load),
((electricity_network, 'P_load2'), load)]
res = master_simulator.simulate(final_time = 86400, input = input_object)
I am getting the following error:
Traceback (most recent call last):
File "C:\Users\kosmy\run_csv_pyfmi.py", line 29, in <module>
res = master_simulator.simulate(final_time = 86400, input = input_object)
File "src\pyfmi\master.pyx", line 1474, in pyfmi.master.Master.simulate
File "src\pyfmi\master.pyx", line 1369, in pyfmi.master.Master.specify_external_input
TypeError: tuple indices must be integers or slices, not tuple
Apparently, I do not give the correct format to the input, but I have not found an example demonstrating the correct format when using the Master.
Does anyone know how can I use the input in this case?

def load(t):
return 10, math.cos(t)
input_object = ([(electricity_network, 'P_load1'),(electricity_network, 'P_load2')],load)
another option is
data = np.transpose(np.vstack((t,u,v)))
input_object = (['InputVarI','InputVarP'],data)
error while creating a 2-tuple as input for model.simulate() of fmu model with pyfmi

Basemap scatterplot (Matplotlib) - Error can't set attribute

I have been trying to add two lists of floats to a scatterplot using the Matplotlib. I have simply created a Basemap with mostly default settings. However, whenever I attempt to run the script I get the error below.
#!/usr/bin/python
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
def main():
wmap = Basemap()
wmap.drawmapboundary(fill_color='aqua')
wmap.fillcontinents(color='coral',lake_color='aqua')
wmap.drawcoastlines()
locationData = open("locationData", "r")
lats = []
lons = []
ASN = []
company = []
fileLine = locationData.readlines()
for i in fileLine:
parsedLine = i.split(",")
lats.append(float(parsedLine[0]))
lons.append(float(parsedLine[1]))
ASN.append(parsedLine[2])
company.append(parsedLine[3])
locationData.close()
x,y = wmap(lats,lons)
wmap.scatter(x,y)
plt.show()
if(__name__ == '__main__'):
main()
I receive the error:
Traceback (most recent call last):
File "mapGen.py", line 51, in <module>
main()
File "mapGen.py", line 45, in main
wmap.scatter(lats,lons)
File "/usr/lib/python3.7/site-packages /mpl_toolkits/basemap/__init__.py", line 563, in with_transform
return plotfunc(self,x,y,*args,**kwargs)
File "/usr/lib/python3.7/site-packages/mpl_toolkits/basemap/__init__.py", line 3253, in scatter
self._restore_hold(ax)
File "/usr/lib/python3.7/site-packages/mpl_toolkits/basemap/__init__.py", line 3229, in _restore_hold
ax._hold = self._tmp_hold
AttributeError: can't set attribute
I have tried changing around the attributes to wmap.scatter() and the variables, but there error still persists and I haven't been able to find documentation on this error. Any ideas?

This specific exception is raised when the attribute you're trying to change is a property that doesn't have a setter. It seems the basemap change is needed, adding a setter would solve the issue.
But you can change wmap.scatter(x,y) to plt.scatter(x, y) to solve it.

Using nipype to apply a FLIRT registration to an additional image error

I am trying to do some registration in python using the nipype package. It worked for basic registration:
from nipype.interfaces import fsl
from nipype.testing import example_data
flt = fsl.FLIRT(bins=640, cost_func='mutualinfo')
flt.inputs.in_file = 'myInput.img'
flt.inputs.reference = 'myReference.img'
flt.inputs.out_file = 'moved_subject.nii'
flt.inputs.out_matrix_file = 'subject_to_template.mat'
res = flt.run()
This yielded a successful registration. However, I am trying to apply this registration transformation to a non-brain image in the same space as the input MRI, using the outputted flt.inputs.out_matrix_file = 'subject_to_template.mat'.
I tried the following:
from nipype.interfaces import fsl
flt = fsl.FLIRT(bins=640, cost_func='mutualinfo')
flt.inputs.in_file = 'myNonBrainImage.img'
flt.inputs.reference = 'myReference.img'
flt.inputs.out_file = 'regNonBrain.nii'
flt.inputs.in_matrix_file = 'subject_to_template.mat'
flt.inputs.apply_xfm = True
res = flt.run()
Hoping that the flt.inputs.in_matrix_file and flt.inputs.apply_xfm = True flag would override standard registration and just use the matrix to register the additional image, but I got this error:
INFO:interface:stderr 2011-08-10T14:59:17.307116:Unrecognised option D
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/bin/python27/lib/python2.7/site-packages/nipype-0.4.1-py2.7.egg/nipype/interfaces/base.py", line 775, in run
runtime = self._run_interface(runtime)
File "/usr/bin/python27/lib/python2.7/site-packages/nipype-0.4.1-py2.7.egg/nipype/interfaces/base.py", line 1050, in _run_interface
self.raise_exception(runtime)
File "/usr/bin/python27/lib/python2.7/site-packages/nipype-0.4.1-py2.7.egg/nipype/interfaces/base.py", line 1027, in raise_exception
raise RuntimeError(message)
RuntimeError: Command:
flirt -in RF8869_3D_XRT_Dose_CT_A.img -ref clo010809T1Gd.img -out regDose.nii -omat /root/Desktop/Test Data/RF8869_3D_XRT_Dose_CT_A_flirt.mat -applyxfm -bins 640 -searchcost mutualinfo -init subject_to_template.mat
Standard output:
Standard error:
Unrecognised option D
Return code: 255
Interface FLIRT failed to run.
Do you know why and how can I solve this?

There is a space in directory/file name containing your images:
/root/Desktop/Test Data
Rename Test Data as Test_Data and it will work