So, i ran my "ml" model on my local windows machine, everything runs smooth, it just takes 48 hour to fully run every process, naturally i ask the company more procesing power to cut times, they give me a linux simulation server to run my models, but for some reason pandas is giving me the next error:
Traceback (most recent call last):
File "/ANACONDATA/Prediccion_de_Fallas/03_Modelos_y_Scripts/testv14.py", line 894, in <module>
dlist[xx][namer2] = np.where((dlist[xx].too_soon == 0),dlist[xx][column].shift(24) , 0)
File "/opt/anaconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3643, in __setitem__
self._setitem_array(key, value)
File "/opt/anaconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3702, in _setitem_array
self._iset_not_inplace(key, value)
File "/opt/anaconda3/lib/python3.9/site-packages/pandas/core/frame.py", line 3721, in _iset_not_inplace
raise ValueError("Columns must be same length as key")
this is the code where i fails (runs ok on windows), tried using pandas 1.3.5, and 1.4.2 same result
features=['AN_Utility_Supply_Press','AN_LPC_ASV_Position',
'AN_Eng_Brg_3Y_Gap',... 200 something list of features]
dlist = {}
turbo= np.unique(dfx2['SAP'])
for xx in (turbo):
dlist[xx]=dfx2.loc[(dfx2['SAP'] == xx)]
for column in dlist[xx][features]:
namer2=[column+'_'+'Lag']
fails here------>dlist[xx][namer2] = np.where((dlist[xx].too_soon == 0),dlist[xx][column].shift(24) , 0)
# namer3=[column+'_'+'Lchg'+"24"]
# dlist[xx][namer3] = np.where((dlist[xx].too_soon == 0),(dlist[xx][column]-dlist[xx][column].shift(24)) , 0)
namer4=[column+'_'+'mean']
dlist[xx][namer4] = np.where((dlist[xx].too_soon == 0),(dlist[xx][column].rolling(min_periods=1, window=feature_window).mean()), dlist[xx][column])
namer5=[column+'_'+'max']
dlist[xx][namer5] = np.where((dlist[xx].too_soon == 0),(dlist[xx][column].rolling(min_periods=1, window=feature_window).max()), dlist[xx][column])
dfx2 = pd.concat(dlist)
dfx2.reset_index(drop=True)
dfx2=dfx2.droplevel(level=0)
am i missing something?, why this happens?
Ok, took some time to figure it out, i tried more versions of pandas until it work, the version is 1.2.4 dont really have an explanation to what happen.
Related
I have a custom geoprocessing task written for me in python that takes 2 inputs - input_rows.csv and event_rows.csv - and creates a file gdb with a feature class of a join of these 2 tables.
input_rows has the x,y coordinates for the feature class.
event_rows.csv contains the attributes.
When I run it locally in ArcCatalog (10.7.1), it runs fine. If I publish it to our ArcGIS 10.81 Linux server and run that in ArcCatalog, it also runs fine. If I try to execute it thru a call in my C# application, I get a generic error on TableToTable_conversion:
<Msg time='2022-10-25T13:01:43,305' type='SEVERE' code='20010' target='export/Download_v3.GPServer' methodName='GPServerSync.CheckMessages' machine='DCP-G-XPLLAGS01.GEOPLAN.UFL.EDU' process='116658' thread='505' user='null' requestId='55b85e40-1ea9-4934-ade7-124fe4dc86c9'>Error executing tool. Download Job ID: j46c9f46dd97642e4890778604255a1fb :
Traceback (most recent call last):
File "<string>", line 248, in execute
File "<string>", line 190, in zip_GIS
File "z:\home\esri\arcgis\server\arcpy\arcpy\conversion.py", line 2272, in TableToTable
raise e
ExecuteError: ERROR 999999: Error executing function.
Failed to execute (TableToTable).
Here is the code to do the conversion:
x_coords = "S4_X"
y_coords = "S4_Y"
out_event_lyr = "event_layer"
v_scratchFolder_ = "%scratchFolder%"
# create Folder
arcpy.CreateFolder_management(v_scratchFolder_, "Geodatabase")
# create File GDB
arcpy.CreateFileGDB_management(x, gdb_name)
wkt = "PROJCS['NAD_1983_HARN_Florida_GDL_Albers',GEOGCS['GCS_North_American_1983_HARN',DATUM['D_North_American_1983_HARN',SPHEROID['GRS_1980',6378137.0,298.257222101]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Albers'],PARAMETER['False_Easting',400000.0],PARAMETER['False_Northing',0.0],PARAMETER['Central_Meridian',-84.0],PARAMETER['Standard_Parallel_1',24.0],PARAMETER['Standard_Parallel_2',31.5],PARAMETER['Latitude_Of_Origin',24.0],UNIT['Meter',1.0],AUTHORITY['EPSG',3087]]"
sr = arcpy.SpatialReference()
sr.loadFromString(wkt)
out_event_lyr = arcpy.MakeXYEventLayer_management(input_rows, x_coords, y_coords, out_event_lyr, sr)
crash_fc = arcpy.CopyFeatures_management(out_event_lyr, os.path.join("%scratchFolder%\Geodatabase", gdb_name, out_event_fc))
event_tab = arcpy.TableToTable_conversion(event_rows, os.path.join("%scratchFolder%\Geodatabase", gdb_name), "EventTab")
crash_fc_lyr = arcpy.MakeFeatureLayer_management(crash_fc, "fc_lyr")
out_event_lyr_join = arcpy.AddJoin_management(crash_fc_lyr, "REPORT_NUMBER", event_tab, "REPORT_NUMBER")
arcpy.CopyFeatures_management(out_event_lyr_join, os.path.join("%scratchFolder%\Geodatabase", gdb_name, out_event_fc1))
We changed the script to do a CopyFeatures instead of TableToTable to see if that would work but I get a different error:
ExecuteError: ERROR 000210: Cannot create output Z:\opt\esri\arcgis\server\usr\directories\arcgisjobs\export\download_v3_1_gpserver\j46744155824f44e1ad5f7c1a1a6afc71\scratch\Geodatabase\s4_mapped_crashes_gdb.gdb\att
Failed to execute (CopyFeatures).
Here is the CopyFeature version:
x_coords = "S4_X"
y_coords = "S4_Y"
out_event_lyr = "event_layer"
v_scratchFolder_ = "%scratchFolder%"
Geodatabase = v_scratchFolder_
s4_crash_gdb_gdb = Geodatabase
# create Folder
x = arcpy.CreateFolder_management(v_scratchFolder_, "Geodatabase")
arcpy.AddMessage(x)
# create File GDB
arcpy.CreateFileGDB_management(x, gdb_name)
wkt = "PROJCS['NAD_1983_HARN_Florida_GDL_Albers',GEOGCS['GCS_North_American_1983_HARN',DATUM['D_North_American_1983_HARN',SPHEROID['G RS_1980',6378137.0,298.257222101]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Albers'],PARAMETER['False_Easting',400000.0],PARAMETER['False_Northing',0.0],PARAMETER['Central_Meridian',-84.0],PARAMETER['Standard_Parallel_1',24.0],PARAMETER['Standard_Parallel_2',31.5],PARAMETER['Latitude_Of_Origin',24.0],UNIT['Meter',1.0],AUTHORITY['EPSG',3087]]"
sr = arcpy.SpatialReference()
sr.loadFromString(wkt)
# create GIS file in geodatabase
out_event_lyr = arcpy.MakeXYEventLayer_management(input_rows, x_coords, y_coords, out_event_lyr, sr)
crash_fc = arcpy.CopyFeatures_management(out_event_lyr, os.path.join("%scratchFolder%\Geodatabase", gdb_name, out_event_fc))
crash_fc_lyr = arcpy.MakeFeatureLayer_management(crash_fc, "fc_lyr")
out_event_lyr_join = arcpy.AddJoin_management(crash_fc_lyr, "REPORT_NUMBER", event_rows, "REPORT_NUMBER")
arcpy.CopyFeatures_management(out_event_lyr_join, os.path.join("%scratchFolder%\Geodatabase", gdb_name, out_event_fc1))
My ArcGIS and Python skills are pretty rudimentary so I'm struggling trying to figure out what the issue is. Doesn't seem like an unreasonable task I'm trying to accomplish but it has me stumped. What am I missing?
I have wrote a code to monitor ram memory usage, CPU memory usage, and CPU temperature I have used both psutil and WMI and I'm some kind of problem the code ran perfectly when I had windows 10 I updated to window 11 it is not working. I have checked python interpreter it is on 3.10
I get this my output/error message:
C:\Users\jeries\PycharmProjects\PP1\venv\Scripts\python.exe C:/Users/jeries/PycharmProjects/PP1/study.py
The CPU usage is: 47.1
RAM memory % used: 54.0
Traceback (most recent call last):
File "C:\Users\jeries\PycharmProjects\PP1\venv\lib\site-packages\wmi.py", line 880, in query
return self._namespace.query(wql, self, fields)
File "C:\Users\jeries\PycharmProjects\PP1\venv\lib\site-packages\wmi.py", line 1072, in query
return [ _wmi_object(obj, instance_of, fields) for obj in self._raw_query(wql) ]
File "C:\Users\jeries\PycharmProjects\PP1\venv\lib\site-packages\wmi.py", line 1072, in <listcomp>
return [ _wmi_object(obj, instance_of, fields) for obj in self._raw_query(wql) ]
File "C:\Users\jeries\PycharmProjects\PP1\venv\lib\site-packages\win32com\client\dynamic.py", line 324, in __getitem__
return self._get_good_object_(self._enum_.__getitem__(index))
File "C:\Users\jeries\PycharmProjects\PP1\venv\lib\site-packages\win32com\client\util.py", line 41, in __getitem__
return self.__GetIndex(index)
File "C:\Users\jeries\PycharmProjects\PP1\venv\lib\site-packages\win32com\client\util.py", line 62, in __GetIndex
result = self._oleobj_.Next(1)
pywintypes.com_error: (-2147217372, 'OLE error 0x80041024', None, None)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\jeries\PycharmProjects\PP1\study.py", line 30, in <module>
temperature_infos = w.Sensor()
File "C:\Users\jeries\PycharmProjects\PP1\venv\lib\site-packages\wmi.py", line 882, in query
handle_com_error()
File "C:\Users\jeries\PycharmProjects\PP1\venv\lib\site-packages\wmi.py", line 258, in handle_com_error
raise klass(com_error=err)
wmi.x_wmi: <x_wmi: Unexpected COM Error (-2147217372, 'OLE error 0x80041024', None, None)>
Process finished with exit code 1
I have tried this:
w = wmi.WMI(namespace="root\openHardwareMonitor")
temperature_infos = w.Sensor()
for sensor in temperature_infos:
if sensor.SensorType == u'Temperature':
print(sensor.Name)
print(sensor.Value)
not working it says that the w.Senosor() "no documentation found"
this is my current code:
import os
import psutil
import wmi
def avg(value_list):
num = 0
length = len(value_list)
for val in value_list:
num += val
return num / length
# Calling psutil.cpu_precent() after 2 seconds
print('The CPU usage is: ', psutil.cpu_percent(2))
print('RAM memory % used:', psutil.virtual_memory()[2])
# have the open hardware monitor opened
w = wmi.WMI(namespace="root\\OpenHardwareMonitor")
sensors = w.Sensor()
cpu_temps = []
gpu_temp = 0
for sensor in sensors:
if sensor.SensorType == u'Temperature' and not 'GPU' in sensor.Name:
cpu_temps += [float(sensor.Value)]
elif sensor.SensorType == u'Temperature' and 'GPU' in sensor.Name:
gpu_temp = sensor.Value
print("Avg CPU: {}".format(avg(cpu_temps)))
print("GPU: {}".format(gpu_temp))
OpenHardwareMonitor can generate logs (options/log Sensors)
The log is called OpenHardwareMonitorLog-YYYY-MM-DD.csv
The idea is therefore to launch OpenHardwareMonitor beforehand (possible to execute via scheduled task + subprocess, or in automatic execution at startup), and to retrieve the correct column in the last line of the file:
#Code
from datetime import date
while 1 == 1:
#Génère le nom du log
now = date.today()
infile = r"C:\OpenHardwareMonitor\OpenHardwareMonitorLog-" + now.strftime("%Y-%m-%d") + ".csv"
#Ouvre en lecture seule
with open(infile, "r") as f:
f = f.readlines()[-1] #Lis la dernière ligne
output = f.split(',') # Sépare via les ","
print(output[10]) # 10 = Colonne T°CPU Core #1
edit:
You will have to find your column number by looking at the log, it's 10 for me, but it must be able to change depending on your config...
I'm just starting, the script should be able to be improved by scanning the first 2 lines and determining the correct column with its name ;-)
I'm trying to write a script to convert an Intel HEX file to a Verilog mem format.
I can print the strings I want to save OK (eg the read & parse bit's working) but when I try to write to a file nothing ever appears :(
ihexf = open("test.hex","r")
vmemf = open("test.mem","w")
for line in ihexf:
rlen_s = line[1:3]
addr_s = line[3:7]
rtyp_s = line[7:9]
rlen = int(rlen_s, 16)
addr = int(addr_s, 16)
rtyp = int(rtyp_s, 16)
# print(rlen_s,addr_s,rtyp_s)
if rtyp == 0:
# print('#'+addr_s)
vmemf.write('#'+addr_s+'\n')
for i in range (0, rlen):
laddr = addr + i
val_s = line[9+i*2:9+i*2+2]
val = int(val_s, 16)
# print(val_s)
vmemf.write(val_s+'\n')
# print("")
else:
print("------- End Of File ------")
ihexf.close()
vmemf.close()
My test.hex looks like
:20000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF000000FF000000FF000000FF555540FF0A
:20000800155540FF055540FF015540FF005540FF001540FF000540FF000140FF000040FF56
:20001000000040FF000140FF000540FF001540FF005540FF015540FF055540FF155540FF4E
:00000001FF
Any clues what I'm doing wrong?
Make sure you have closed the file and very importantly that you reposition the file pointer to the start of the file and start reading chunks.
ihexf.seek(0,0)
OK - I worked out what was happening (I think!)
Existing code works on linux but not Windows.
On Windows I was seeing the following once the script finished:
#0000
#0008
#0010
#0018
------- End Of File ------
Traceback (most recent call last):
File "C:\Users\Nigel\SkyDrive\Files\python\intexhex2v.py", line 8, in <module>
rlen = int(rlen_s, 16)
ValueError: invalid literal for int() with base 16: ''`
Looks like things were messing up at the end of the file read.
Adding break after the End-Of-File print fixed everything
Thanks
I am trying to execute this script:
import time
from SECEdgar.crawler import SecCrawler
def get_filings():
t1 = time.time()
# create object
seccrawler = SecCrawler()
companyCode = 'AAPL' # company code for apple
cik = '0000320193' # cik code for apple
date = '20010101' # date from which filings should be downloaded
count = '10' # no of filings
seccrawler.filing_10Q(str(companyCode), str(cik), str(date), str(count))
seccrawler.filing_10K(str(companyCode), str(cik), str(date), str(count))
seccrawler.filing_8K(str(companyCode), str(cik), str(date), str(count))
seccrawler.filing_13F(str(companyCode), str(cik), str(date), str(count))
t2 = time.time()
print "Total Time taken: ",
print (t2-t1)
if __name__ == '__main__':
get_filings()
I am putting this code in a file filings.py , then attempt to run it from terminal (Mac user)
python filings.py
But I am getting the following error:
Traceback (most recent call last):
File "filings.py", line 2, in <module>
from SECEdgar.crawler import SecCrawler
File "build/bdist.macosx-10.10-intel/egg/SECEdgar/crawler.py", line 6, in <module>
File "build/bdist.macosx-10.10-intel/egg/SECEdgar/config.py", line 22, in <module>
File "/Library/Python/2.7/site-packages/configparser.py", line 995, in __getitem__
raise KeyError(key)
KeyError: 'Paths'
What am I doing wrong?
Looks like there's an error in the package you installed.
Try uninstalling and reinstalling.
pip uninstall SECEdgar
pip install SECEdgar
I found the solution, it was basically a quite silly thing:
date = '20010101' # date from which filings should be downloaded
should be
date = '20010101' # date UNTIL which filings should be downloaded
so if you put the starting date you would end up with download 0 files, but if you put the end date then you would download them all successfully, seems to be working OK now :)
I am trying to do some registration in python using the nipype package. It worked for basic registration:
from nipype.interfaces import fsl
from nipype.testing import example_data
flt = fsl.FLIRT(bins=640, cost_func='mutualinfo')
flt.inputs.in_file = 'myInput.img'
flt.inputs.reference = 'myReference.img'
flt.inputs.out_file = 'moved_subject.nii'
flt.inputs.out_matrix_file = 'subject_to_template.mat'
res = flt.run()
This yielded a successful registration. However, I am trying to apply this registration transformation to a non-brain image in the same space as the input MRI, using the outputted flt.inputs.out_matrix_file = 'subject_to_template.mat'.
I tried the following:
from nipype.interfaces import fsl
flt = fsl.FLIRT(bins=640, cost_func='mutualinfo')
flt.inputs.in_file = 'myNonBrainImage.img'
flt.inputs.reference = 'myReference.img'
flt.inputs.out_file = 'regNonBrain.nii'
flt.inputs.in_matrix_file = 'subject_to_template.mat'
flt.inputs.apply_xfm = True
res = flt.run()
Hoping that the flt.inputs.in_matrix_file and flt.inputs.apply_xfm = True flag would override standard registration and just use the matrix to register the additional image, but I got this error:
INFO:interface:stderr 2011-08-10T14:59:17.307116:Unrecognised option D
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/bin/python27/lib/python2.7/site-packages/nipype-0.4.1-py2.7.egg/nipype/interfaces/base.py", line 775, in run
runtime = self._run_interface(runtime)
File "/usr/bin/python27/lib/python2.7/site-packages/nipype-0.4.1-py2.7.egg/nipype/interfaces/base.py", line 1050, in _run_interface
self.raise_exception(runtime)
File "/usr/bin/python27/lib/python2.7/site-packages/nipype-0.4.1-py2.7.egg/nipype/interfaces/base.py", line 1027, in raise_exception
raise RuntimeError(message)
RuntimeError: Command:
flirt -in RF8869_3D_XRT_Dose_CT_A.img -ref clo010809T1Gd.img -out regDose.nii -omat /root/Desktop/Test Data/RF8869_3D_XRT_Dose_CT_A_flirt.mat -applyxfm -bins 640 -searchcost mutualinfo -init subject_to_template.mat
Standard output:
Standard error:
Unrecognised option D
Return code: 255
Interface FLIRT failed to run.
Do you know why and how can I solve this?
There is a space in directory/file name containing your images:
/root/Desktop/Test Data
Rename Test Data as Test_Data and it will work