So I am running pytest via github actions and a test that passes on my machine (both through direct command line and vscode testing) is erroring out and aborting in the container and idk why.
============================= test session starts ==============================
platform linux -- Python 3.10.10, pytest-7.2.1, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.10.10/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/jbuielCubesProject/jbuielCubesProject
collecting ... collected 4 items
tests/test_data.py::test_api_data_amount PASSED [ 25%]
tests/test_data.py::test_entry_in_database PASSED [ 50%]
tests/test_data.py::test_data_in_table PASSED [ 75%]
Fatal Python error: Aborted
Current thread 0x00007f6d807b6c40 (most recent call first):
File "/home/runner/work/jbuielCubesProject/jbuielCubesProject/tests/test_data.py", line 55 in test_gui_info
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/python.py", line 1789 in runtest
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/runner.py", line 260 in <lambda>
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/runner.py", line 339 in from_call
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/runner.py", line 220 in call_and_report
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/runner.py", line 131 in runtestprotocol
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/main.py", line 324 in _main
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/main.py", line 270 in wrap_session
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/config/__init__.py", line 167 in main
File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/site-packages/_pytest/config/__init__.py", line 190 in console_main
File "/opt/hostedtoolcache/Python/3.10.10/x64/bin/pytest", line 8 in <module>
Extension modules: charset_normalizer.md, xxsubtype, shiboken6.Shiboken, PySide6.QtCore, PySide6.QtGui, PySide6.QtWidgets (total: 6)
/home/runner/work/_temp/835fb4a2-07d9-4169-ac69-9365002eaa42.sh: line 2: 1885 Aborted (core dumped) pytest -v
tests/test_data.py::test_gui_info
Error: Process completed with exit code 134.
I don't see anything obvious in this traceback that would help me fix it, anyone know what to do from here?
I thought 134 was a memory error so i tried adding in a swap space from https://github.com/pierotofy/set-swap-space but that still didn't help
def test_gui_info():
# test 4 sprint 3
QtWidgets.QApplication([]) # line 55 <-------
MainWindow = QtWidgets.QMainWindow()
ui = database_viewer(MainWindow)
labelled_entries_dict = get_labelled_dict()
ui.show_data(id=2, labelled_entries_dict=labelled_entries_dict)
response = ui.rightLayout.itemAt(3).widget()
assert response.text() == labelled_entries_dict[2][' First Name']
response = ui.rightLayout.itemAt(5).widget()
assert response.text() == labelled_entries_dict[2][' Last Name']
response = ui.rightLayout.itemAt(11).widget()
assert response.text() == labelled_entries_dict[2][' Email']
response = ui.rightLayout.itemAt(13).widget()
assert response.text() == labelled_entries_dict[2][' Organization Website']
response = ui.rightLayout.itemAt(19).widget()
if labelled_entries_dict[2][' Course Project'] == 'yes':
assert response.isChecked() is True
else:
assert response.isChecked() is False
response = ui.rightLayout.itemAt(20).widget()
if labelled_entries_dict[2][' Guest Speaker'] == 'yes':
assert response.isChecked() is True
else:
assert response.isChecked() is False
response = ui.rightLayout.itemAt(23).widget()
if labelled_entries_dict[2][' Internships'] == 'yes':
assert response.isChecked() is True
else:
assert response.isChecked() is False
response = ui.rightLayout.itemAt(30).widget()
if labelled_entries_dict[2][' Summer 2023'] == 'yes':
assert response.isChecked() is True
else:
assert response.isChecked() is False
after a lot of trial and error I found that adding these to the GHA made it work
under runs-on: ubuntu-latest
env:
DISPLAY: ':99.0'
then anywhere in the steps:
- uses: tlambert03/setup-qt-libs#v1
- name: build "display"
run: |
/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -screen 0 1920x1200x24 -ac +extension GLX
im not actually using pytest-qt but this troubleshooting page gave these answer, it was very hard to find until I stumbled upon it eventually.
https://pytest-qt.readthedocs.io/en/latest/troubleshooting.html
Related
The second time I play the program in the console, it has a 'fatal error'.
I have been solving this by moving to a new console, but I'm still worried that this could cause problems later.
Code:
import pygame
from pygame.locals import *
board = pygame.display.set_mode((400,600))
#colors
black = pygame.Color(0, 0, 0)
white = pygame.Color(255, 255, 255)
grey = pygame.Color(128, 128, 128)
red = pygame.Color(255, 0, 0)
blue = pygame.Color(0, 0, 255)
green=pygame.Color(0,255,0)
board.fill(white)
pygame.display.set_caption("Sigil")
class Card (pygame.sprite.Sprite):
def __init__(self,image,x,y):
super().__init__()
#above is magic class code that shouldn't be messed with (mostly)
#inuit only happens once
self.x=x
self.y=y
self.image=image
self.rect = self.image.get_rect()
board.blit(self.image,(self.x,self.y))
def update(self):
if event.type == pygame.KEYDOWN:
if event.key == pygame.K_w:
self.rect.move_ip(0, -5)
elif event.key == pygame.K_s:
self.rect.move_ip(0,5)
elif event.key == pygame.K_a:
self.rect.move_ip(-5, 0)
elif event.key == pygame.K_d:
self.rect.move_ip(5, 0)
def draw(self, surface):
surface.blit(self.image, self.rect)
test=pygame.image.load("Twitter-icon.png") #Make sure to add .png
test=pygame.transform.scale(test,(test.get_width()/2,test.get_height()/2))
P1 = Card(test,10,10)
print_message=pygame.USEREVENT + 0
pygame.time.set_timer(print_message, 3000) #can be used to make longer loops for your loop
#Frames Per Second
clock=pygame.time.Clock()
#Game Loop
while True:
#All functions dependent on user input need to be part of this loop- event.get clears the event que
for event in pygame.event.get():
if event.type == QUIT:
pygame.quit()
if event.type == print_message:
print("Hello World")
P1.update()
board.fill(white)
P1.draw(board)
pygame.display.flip()
clock.tick(20)
Here's the error
Fatal Python error: (pygame parachute) Segmentation Fault
Thread 0x00008390 (most recent call first):
File "C:\Users\HP\anaconda3\lib\site‑packages\ipykernel\parentpoller.py", line 97 in run
File "C:\Users\HP\anaconda3\lib\threading.py", line 926 in _bootstrap_inner
File "C:\Users\HP\anaconda3\lib\threading.py", line 890 in _bootstrap
Thread 0x0000586c (most recent call first):
File "C:\Users\HP\anaconda3\lib\site‑packages\zmq\sugar\socket.py", line 475 in
recv_multipart
File "C:\Users\HP\anaconda3\lib\site‑packages\jupyter_client\session.py", line 803 in recv
File "C:\Users\HP\anaconda3\lib\site‑packages\spyder_kernels\comms\frontendcomm.py", line
114 in poll_one
File "C:\Users\HP\anaconda3\lib\site‑packages\spyder_kernels\comms\frontendcomm.py", line
104 in poll_thread
File "C:\Users\HP\anaconda3\lib\threading.py", line 870 in run
File "C:\Users\HP\anaconda3\lib\threading.py", line 926 in _bootstrap_inner
File "C:\Users\HP\anaconda3\lib\threading.py", line 890 in _bootstrap
Thread 0x00005f14 (most recent call first):
File "C:\Users\HP\anaconda3\lib\threading.py", line 296 in wait
File "C:\Users\HP\anaconda3\lib\threading.py", line 552 in wait
File "C:\Users\HP\anaconda3\lib\site‑packages\IPython\core\history.py", line 829 in run
File "C:\Users\HP\anaconda3\lib\site‑packages\IPython\core\history.py", line 58 in
needs_sqlite
File "", line 2 in run
File "C:\Users\HP\anaconda3\lib\threading.py", line 926 in _bootstrap_inner
File "C:\Users\HP\anaconda3\lib\threading.py", line 890 in _bootstrap
Thread 0x00006c64 (most recent call first):
File "C:\Users\HP\anaconda3\lib\site‑packages\ipykernel\heartbeat.py", line 100 in run
File "C:\Users\HP\anaconda3\lib\threading.py", line 926 in _bootstrap_inner
File "C:\Users\HP\anaconda3\lib\threading.py", line 890 in _bootstrap
Thread 0x00006558 (most recent call first):
File "C:\Users\HP\anaconda3\lib\selectors.py", line 314 in _select
File "C:\Users\HP\anaconda3\lib\selectors.py", line 323 in select
File "C:\Users\HP\anaconda3\lib\asyncio\base_events.py", line 1746 in _run_once
File "C:\Users\HP\anaconda3\lib\asyncio\base_events.py", line 538 in run_forever
File "C:\Users\HP\anaconda3\lib\site‑packages\tornado\platform\asyncio.py", line 153 in
start
File "C:\Users\HP\anaconda3\lib\site‑packages\ipykernel\iostream.py", line 78 in _
thread_main
File "C:\Users\HP\anaconda3\lib\threading.py", line 870 in run
File "C:\Users\HP\anaconda3\lib\threading.py", line 926 in _bootstrap_inner
File "C:\Users\HP\anaconda3\lib\threading.py", line 890 in _bootstrap
Current thread 0x000060c8 (most recent call first):
File "C:\Users\HP\Documents\Python Scripts\Cardgame(sigil)prototype.py", line 4 in
File
"C:\Users\HP\anaconda3\lib\site‑packages\spyder_kernels\customize\spydercustomize.py",
line 377 in exec_code
File
"C:\Users\HP\anaconda3\lib\site‑packages\spyder_kernels\customize\spydercustomize.py",
line 476 in runfile
File "", line 1 in
File "C:\Users\HP\anaconda3\lib\site‑packages\IPython\core\interactiveshell.py", line 3331
in run_code
File "C:\Users\HP\anaconda3\lib\site‑packages\IPython\core\interactiveshell.py", line 3254
in run_ast_nodes
File "C:\Users\HP\anaconda3\lib\site‑packages\IPython\core\interactiveshell.py", line 3063
in run_cell_async
File "C:\Users\HP\anaconda3\lib\site‑packages\IPython\core\async_helpers.py", line 68 in
_pseudo_sync_runner
File "C:\Users\HP\anaconda3\lib\site‑packages\IPython\core\interactiveshell.py", line 2886
in _run_cell
File "C:\Users\HP\anaconda3\lib\site‑packages\IPython\core\interactiveshell.py", line 2858
in run_cell
File "C:\Users\HP\anaconda3\lib\site‑packages\ipykernel\zmqshell.py", line 536 in run_cell
File "C:\Users\HP\anaconda3\lib\site‑packages\ipykernel\ipkernel.py", line 300 in
do_execute
File "C:\Users\HP\anaconda3\lib\site‑packages\tornado\gen.py", line 209 in wrapper
File "C:\Users\HP\anaconda3\lib\site‑packages\ipykernel\kernelbase.py", line 541 in
execute_request
File "C:\Users\HP\anaconda3\lib\site‑packages\tornado\gen.py", line 209 in wrapper
File "C:\Users\HP\anaconda3\lib\site‑packages\ipykernel\kernelbase.py", line 268 in
dispatch_shell
File "C:\Users\HP\anaconda3\lib\site‑packages\tornado\gen.py", line 209 in wrapper
File "C:\Users\HP\anaconda3\lib\site‑packages\ipykernel\kernelbase.py", line 361 in
process_one
File "C:\Users\HP\anaconda3\lib\site‑packages\tornado\gen.py", line 748 in run
File "C:\Users\HP\anaconda3\lib\site‑packages\tornado\gen.py", line 787 in inner
File "C:\Users\HP\anaconda3\lib\site‑packages\tornado\ioloop.py", line 743 in
_run_callback
File "C:\Users\HP\anaconda3\lib\site‑packages\tornado\ioloop.py", line 690 in
File "C:\Users\HP\anaconda3\lib\asyncio\events.py", line 88 in _run
File "C:\Users\HP\anaconda3\lib\asyncio\base_events.py", line 1782 in _run_once
File "C:\Users\HP\anaconda3\lib\asyncio\base_events.py", line 538 in run_forever
File "C:\Users\HP\anaconda3\lib\site‑packages\tornado\platform\asyncio.py", line 153 in
start
File "C:\Users\HP\anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 583 in start
File "C:\Users\HP\anaconda3\lib\site‑packages\spyder_kernels\console\start.py", line 318
in main
File "C:\Users\HP\anaconda3\lib\site‑packages\spyder_kernels\console\__main__.py", line 11
in
File "C:\Users\HP\anaconda3\lib\runpy.py", line 85 in _run_code
File "C:\Users\HP\anaconda3\lib\runpy.py", line 193 in _run_module_as_main
Due to my lack of experience, my state of understanding is "WTF" due to me realizing (while inputting) that this error is longer than my entire program.
IDE's like spyder tend to run your code differently than when you run it from a cmd window.
So if your program crashes or behaves weird in an IDE, the first thing to try is to save it as a file and run it from a cmd window as python filename.py and see if that works as intended.
If it does, mark it down as an IDE bug/feature.
I would say that this is particularly relevant for programs that use graphics toolkits like pygame or tkinter. Or programs that use multiprocessing, threading or subprocess.
I'm running batch-wise predictions on a pre-trained model. The error is raised during the dataset creation process, specifically in a tf.data.Dataset.map() function.
This is the offending function-
paths_dataset = tf.data.Dataset.from_tensor_slices(file_list)
# #tf.autograph.experimental.do_not_convert
def path_to_wav(path):
audio_bytes = tf.io.read_file(path)
wav, sr = tf.audio.decode_wav(audio_bytes, desired_channels=1, desired_samples=example_sample_length)
wav = tfio.audio.resample(wav, rate_in=tf.cast(sr, dtype=tf.int64), rate_out=resample_rate)
return tf.transpose(wav, perm=[1, 0])
wavs_dataset = paths_dataset.map(path_to_wav)
It has been tough to debug because the script works fine in a debugger and in a notebook. It only throws an error if I run the script via terminal or send the job to a cluster node-
WARNING:tensorflow:AutoGraph could not transform <function resample at 0x2b0dc6249160> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: std::bad_cast
To silence this warning, decorate the function with #tf.autograph.experimental.do_not_convert
srun: error: node416: task 0: Segmentation fault
Adding the #tf.autograph.experimental.do_not_convert decorator suppresses the warning but throws the following error-
File "/home/s/ss645/mlos/train/estimates_dataset.py", line 120, in simple_dataset
wavs_dataset = paths_dataset.map(path_to_wav)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2048, in map
return MapDataset(self, map_func, preserve_cardinality=True, name=name)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 5243, in __init__
self._map_func = structured_function.StructuredFunctionWrapper(
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/structured_function.py", line 271, in __init__
self._function = fn_factory()
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2567, in get_concrete_function
graph_function = self._get_concrete_function_garbage_collected(
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2533, in _get_concrete_function_garbage_collected
graph_function, _ = self._maybe_define_function(args, kwargs)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2711, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2627, in _create_graph_function
func_graph_module.func_graph_from_py_func(
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py", line 1141, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/structured_function.py", line 248, in wrapped_fn
ret = wrapper_helper(*args)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/structured_function.py", line 177, in wrapper_helper
ret = autograph.tf_convert(self._func, ag_ctx)(*nested_args)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 642, in wrapper
return func(*args, **kwargs)
File "/home/s/ss645/mlos/train/estimates_dataset.py", line 115, in path_to_wav
wav = tfio.audio.resample(wav, rate_in=sr, rate_out=resample_rate)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/audio_ops.py", line 462, in resample
value = tf.vectorized_map(f, input)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 549, in vectorized_map
return pfor(loop_fn, batch_size,
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 206, in pfor
outputs = f()
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 187, in f
return _pfor_impl(loop_fn,
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 286, in _pfor_impl
loop_fn_outputs = f(loop_var)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 642, in wrapper
return func(*args, **kwargs)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 530, in loop_fn
return fn(gathered_elems)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/audio_ops.py", line 458, in f
return core_ops.io_audio_resample(
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py", line 88, in __getattr__
return getattr(self._load(), attrb)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py", line 84, in _load
self._mod = _load_library(self._library)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py", line 64, in _load_library
l = load_fn(f)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
RuntimeError: std::bad_cast
srun: error: node650: task 0: Exited with exit code 1
Any help would be much appreciated! I'm currently unable to file a Tensorflow issue because the error is not reproducible in a notebook.
I have a code like below.
shapes_list = [shapely.wkt.loads(entry['shape'] if type(entry) is dict else entry) for entry in entries]
The problem is that tracemalloc points to this place as the memory leak. Has anyone encountered this?
EDIT logs:
*** Trace for largest memory block - (102367 blocks, 12395.703125 Kb) ***
File "file.py", line 27
masks = obj.generate_masks()
File "/code/src/project/processing/file.py", line 514
image_objects_dict=image_objects_dict)
File "/code/src/project/processing/mask.py", line 322
shapes_list = [shapely.wkt.loads(entry['shape'] if type(entry) is dict else entry) for entry in entries]
File "/code/src/project/processing/mask.py", line 322
shapes_list = [shapely.wkt.loads(entry['shape'] if type(entry) is dict else entry) for entry in entries]
File "/code/venv/lib/python3.7/site-packages/shapely/wkt.py", line 22
return geos.WKTReader(geos.lgeos).read(data)
File "/code/venv/lib/python3.7/site-packages/shapely/geos.py", line 337
return geom_factory(geom)
File "/code/venv/lib/python3.7/site-packages/shapely/geometry/base.py", line 84
ob._set_geom(g)
File "/code/venv/lib/python3.7/site-packages/shapely/geometry/base.py", line 241
self._empty()
File "/code/venv/lib/python3.7/site-packages/shapely/geometry/base.py", line 199
self._is_empty = True
File "/code/venv/lib/python3.7/site-packages/shapely/geometry/base.py", line 250
super().__setattr__(name, value)
I am writing a bootstrap algorithm using parallel loops and pandas. The problem i experience is that a merge command inside the parallel loop causes a "ValueError: buffer source array is read-only" error - but only if i use the full dataset to merge (120k lines). Any subset with less than 12k lines will work just fine and so i infer it is not a problem of the syntax. What can i do?
Current pandas version is 0.24.2 and cython is 0.29.7.
_RemoteTraceback Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.6/site-packages/joblib/externals/loky/process_executor.py", line 418, in _process_worker
r = call_item()
File "/home/ubuntu/.local/lib/python3.6/site-packages/joblib/externals/loky/process_executor.py", line 272, in __call__
return self.fn(*self.args, **self.kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 567, in __call__
return self.func(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/joblib/parallel.py", line 225, in __call__
for func, args, kwargs in self.items]
File "/home/ubuntu/.local/lib/python3.6/site-packages/joblib/parallel.py", line 225, in <listcomp>
for func, args, kwargs in self.items]
File "<ipython-input-72-cdb83eaf594c>", line 12, in bootstrap
File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/frame.py", line 6868, in merge
copy=copy, indicator=indicator, validate=validate)
File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 48, in merge
return op.get_result()
File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 546, in get_result
join_index, left_indexer, right_indexer = self._get_join_info()
File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 756, in _get_join_info
right_indexer) = self._get_join_indexers()
File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 735, in _get_join_indexers
how=self.how)
File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 1130, in _get_join_indexers
llab, rlab, shape = map(list, zip(* map(fkeys, left_keys, right_keys)))
File "/home/ubuntu/.local/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 1662, in _factorize_keys
rlab = rizer.factorize(rk)
File "pandas/_libs/hashtable.pyx", line 111, in pandas._libs.hashtable.Int64Factorizer.factorize
File "stringsource", line 653, in View.MemoryView.memoryview_cwrapper
File "stringsource", line 348, in View.MemoryView.memoryview.__cinit__
ValueError: buffer source array is read-only
"""
The above exception was the direct cause of the following exception:
ValueError Traceback (most recent call last)
<ipython-input-73-652c1db5701b> in <module>()
1 num_cores = multiprocessing.cpu_count()
----> 2 results = Parallel(n_jobs=num_cores, prefer='processes', verbose = 5)(delayed(bootstrap)() for i in range(n_trials))
3 #pd.DataFrame(results[0])
~/.local/lib/python3.6/site-packages/joblib/parallel.py in __call__(self, iterable)
932
933 with self._backend.retrieval_context():
--> 934 self.retrieve()
935 # Make sure that we get a last message telling us we are done
936 elapsed_time = time.time() - self._start_time
~/.local/lib/python3.6/site-packages/joblib/parallel.py in retrieve(self)
831 try:
832 if getattr(self._backend, 'supports_timeout', False):
--> 833 self._output.extend(job.get(timeout=self.timeout))
834 else:
835 self._output.extend(job.get())
~/.local/lib/python3.6/site-packages/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
519 AsyncResults.get from multiprocessing."""
520 try:
--> 521 return future.result(timeout=timeout)
522 except LokyTimeoutError:
523 raise TimeoutError()
/usr/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
430 raise CancelledError()
431 elif self._state == FINISHED:
--> 432 return self.__get_result()
433 else:
434 raise TimeoutError()
/usr/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
382 def __get_result(self):
383 if self._exception:
--> 384 raise self._exception
385 else:
386 return self._result
ValueError: buffer source array is read-only
and the code is
def bootstrap():
df_resample_ids = skl.utils.resample(ob_ids)
df_resample_ids = pd.DataFrame(df_resample_ids).sort_values(by="0").reset_index(drop=True)
df_resample_ids.columns = [ob_id_field]
df_resample = pd.DataFrame(df_resample_ids.merge(df, on = ob_id_field))
return df_resample
num_cores = multiprocessing.cpu_count()
results = Parallel(n_jobs=num_cores, prefer='processes', verbose = 5)(delayed(bootstrap)() for i in range(n_trials))
The algo will create resampled/replaced IDs from an ID variable and use the merge command to create a new dataset based on the resampled IDs and the original dataset stored in df. If i cut out a subset of the original dataset (anywhere) leaving less than ~12k lines, then the parallel loop will finish without an error and do as expected.
As requested, below is a new snippet to re-create the data structures and mirror the principal approach i am currently working on:
import pandas as pd
import sklearn as skl
import multiprocessing
from joblib import Parallel, delayed
df = pd.DataFrame(np.random.randn(200000, 24), columns=list('ABCDDEFGHIJKLMNOPQRSTUVW'))
df["ID"] = df.index.drop_duplicates().tolist()
ob_ids = df.index.drop_duplicates().tolist()
def bootstrap2():
df_resample_ids = skl.utils.resample(ob_ids)
df_resample_ids = pd.DataFrame(df_resample_ids).sort_values(by=0).reset_index(drop=True)
df_resample_ids.columns = ['ID']
df_resample = pd.DataFrame(df1.merge(df_resample_ids, on = 'ID'))
result = df_resample
return result
num_cores = multiprocessing.cpu_count()
results = Parallel(n_jobs=num_cores, prefer='processes', verbose = 5)(delayed(bootstrap2)() for i in range(n_trials))
However, i notice that when the data is completely made up of np.random numbers, the loop goes through without an error. The dtypes of the original dataframe are:
start_rtg int64
end_rtg float64
days_diff float64
ultimate_customer_system_id int64
How can i avoid the read-only error?
posting an answer to my question as i found that one of the variables was of int64 datatype. when i converted all variables to float64, the error disappeared. so it is an issue that is restricted to certain datatypes only...
cheers
stephan
I am trying to run a very simple test using django:
class PaymentTests(TestCase):
def setUp(self):
self.c = Client()
def test_can_see_chat_for_an_anynomous_user(self):
response = self.c.get('/')
self.assertEqual(200, response.status_code)
Unfortunately each time I try to run it I get an error:
(project)linoor#linoor-virtual-machine /mnt/hgfs/Dev/LegalStart/project $ python manage.py test --rednose useracts.tests:PaymentTests
nosetests useracts.tests:PaymentTests --rednose --verbosity=1
Creating test database for alias 'default'...
ERROR 2015-02-11 16:37:00,082 Internal Server Error: /
Traceback (most recent call last):
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 105, in get_response
response = middleware_method(request, callback, callback_args, callback_kwargs)
File "/mnt/hgfs/Dev/LegalStart/project/staticpages/middleware.py", line 56, in process_view
r = Redirect.objects.get(old_path='/' + slug)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/manager.py", line 151, in get
return self.get_queryset().get(*args, **kwargs)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 304, in get
num = len(clone)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 77, in __len__
self._fetch_all()
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 857, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
ProgrammingError: relation "django_redirect" does not exist
LINE 1: ...ct"."old_path", "django_redirect"."new_path" FROM "django_re...
^
X
-----------------------------------------------------------------------------
1) ERROR: test_can_see_chat_for_an_anynomous_user (useracts.tests.PaymentTests)
Traceback (most recent call last):
useracts/tests.py line 48 in test_can_see_chat_for_an_anynomous_user
response = self.c.post('/')
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/test/client.py line 483 in post
response = super(Client, self).post(path, data=data, content_type=content_type, **extra)
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/test/client.py line 302 in post
return self.request(**r)
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/test/client.py line 444 in request
six.reraise(*exc_info)
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/core/handlers/base.py line 105 in get_response
response = middleware_method(request, callback, callback_args, callback_kwargs)
staticpages/middleware.py line 56 in process_view
r = Redirect.objects.get(old_path='/' + slug)
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/manager.py line 151 in get
return self.get_queryset().get(*args, **kwargs)
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py line 304 in get
num = len(clone)
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py line 77 in __len__
self._fetch_all()
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py line 857 in _fetch_all
self._result_cache = list(self.iterator())
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py line 220 in iterator
for row in compiler.results_iter():
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py line 713 in results_iter
for rows in self.execute_sql(MULTI):
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py line 786 in execute_sql
cursor.execute(sql, params)
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/backends/util.py line 53 in execute
return self.cursor.execute(sql, params)
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/utils.py line 99 in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/backends/util.py line 53 in execute
return self.cursor.execute(sql, params)
ProgrammingError: relation "django_redirect" does not exist
LINE 1: ...ct"."old_path", "django_redirect"."new_path" FROM "django_re...
^
-------------------- >> begin captured logging << --------------------
django.request: ERROR: Internal Server Error: /
Traceback (most recent call last):
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 105, in get_response
response = middleware_method(request, callback, callback_args, callback_kwargs)
File "/mnt/hgfs/Dev/LegalStart/project/staticpages/middleware.py", line 56, in process_view
r = Redirect.objects.get(old_path='/' + slug)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/manager.py", line 151, in get
return self.get_queryset().get(*args, **kwargs)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 304, in get
num = len(clone)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 77, in __len__
self._fetch_all()
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 857, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/linoor/virtualenvs/LegalStart/project/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
ProgrammingError: relation "django_redirect" does not exist
LINE 1: ...ct"."old_path", "django_redirect"."new_path" FROM "django_re...
^
--------------------- >> end captured logging << ---------------------
I've tried doing the same thing in the django shell and it works fine.
The project uses South and I found somewhere that I should add these lines to the project.py file:
SOUTH_TESTS_MIGRATE = False
SKIP_SOUTH_TESTS = True
But it still doesn't work.
Do you guys have any idea what might be wrong? I can provide additional info if needed.
Thanks!
Ok, so the problem was Django Mezzanine addon which was dynamically removing the "django.contrib.redirects" from the INSTALLED_APPS.