Develop Reference

Problem with hdbscan used with bertopic: OSError: [Errno 22] Invalid argument

I am writing because I have a problem (silly and obvious introduction, I know).
I am trying to use the BERTopic package using the Python interpreter in RStudio and the reticulate extension:
Python 3.6.13 (C:/Users/Francesco/AppData/Local/r-miniconda/envs/r-reticulate/python.exe)
Reticulate 1.18.9008 REPL -- A Python interpreter in R.
I managed to install it with
pip3 install bertopic
At first, trying to install bertopic resulted in an error realating to its hdbscan dependence, specifically to the wheel used; I overcame it by installing hdbscan by conda (with pip the problem appeared unsolvable) and after doing it seemed that both were installed and fine (pip would confirm so).
Afterwards, I tried to follow the package tutorial in Medium/Towards Data Science (here the Colab version I’m following) to get accostumed with the package and to check that everything was working as supposed to.
I am basically copying and pasting the code of Colab on the Python chunks in the RMarkdown file I am using, but when I try to apply the same code of the tutorial to the same dataset used:
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data']
from bertopic import BERTopic
topic_model = BERTopic(language="english", calculate_probabilities=True, verbose=True)
topics, probs = topic_model.fit_transform(docs)
I get the following error:
Batches: 100%|##########| 589/589 [28:21<00:00, 2.89s/it]
2021-04-29 16:24:25,973 - BERTopic - Transformed documents to Embeddings
2021-04-29 16:24:35,752 - BERTopic - Reduced dimensionality with UMAP
OSError: [Errno 22] Invalid argument
In theory, following the output on colab, I should get:
....................... - BERTopic - Clustered UMAP embeddings with HDBSCAN
Since I had problem with hdbscan I do believe it is somehow related to it, and I read several GitHub and Stackoverflow pages pointing out problems with such a package, but I do not know how to solve this, but I really need to since I need to use package for my thesis.
Can someone help me, please?
PS: it's the first time I am asking stuff on stackoverflow: I hoped I wrote down everything necessary, but if some info is missing, please tell me.

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED using pytorch

I am trying to run a simple pytorch sample code. It's works fine using CPU. But when using GPU, i get this error message:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 263, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 260, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
The code i am trying to run is the following:
import torch
from torch import nn
m = nn.Conv1d(16, 33, 3, stride=2)
m=m.to('cuda')
input = torch.randn(20, 16, 50)
input=input.to('cuda')
output = m(input)
I am running this code in a NVIDIA docker with CUDA version 10.2 and my GPU is a RTX 2070

There is some discussion regarding this here. I had the same issue but using cuda 11.1 resolved it for me.
This is the exact pip command
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

In my case it actually had nothing do with the PyTorch/CUDA/cuDNN version. PyTorch initializes cuDNN lazily whenever a convolution is executed for the first time. However, in my case there was not enough GPU memory left to initialize cuDNN because PyTorch itself already held the entire memory in its internal cache. One can release the cache manually with "torch.cuda.empty_cache()" right before the first convolution that is executed. A cleaner solution is to force cuDNN initialization at the beginning by doing a mock convolution:
def force_cudnn_initialization():
s = 32
dev = torch.device('cuda')
torch.nn.functional.conv2d(torch.zeros(s, s, s, s, device=dev), torch.zeros(s, s, s, s, device=dev))
Calling the above function at the very beginning of the program solved the problem for me.

I am also using Cuda 10.2. I had the exact same error when upgrading torch and torchvision to the latest version (torch-1.8.0 and torchvision-0.9.0). Which version are you using?
I guess this is not the best solution but by downgrading to torch-1.7.1 and torchvision-0.8.2 it works just fine.

In my cases this error occurred when trying to estimate loss.
I used a mixed bce-dice loss.
It turned out that my output was linear instead of sigmoid.
I then used the sigmoid predictions as of bellow and worked fine.
output = torch.nn.Sigmoid()(output)
loss = criterion1(output, target)

I had the same issue when I was training yolov7 with a chess dataset. By reducing batch size from 8 to 4, the issue was solved.

In my problem
i used to kill exisiting process in gpu.Use nvidia-smi to check what are the process are running.Use killall -9 python3(what process you want) to kill process.After freeup space then run the process.

In my case, I had an array indexing operation but the index was out of bounds. CUDA did not tell me that. I was using inference on a neural network. So I moved to CPU instead of the GPU. The logs were much more informative after that. For debugging if you see this error, switch to CPU first and you will know what to do.

MXNet ML lib C++ segmentation fault on OS X

I have a problem with Apache MXNet machine learning library on OS X.
I have been able to run Python version of Lenet, convolutional neural network.
I installed these with pip under both Anaconda Python 2.7 and 3.6.
conda create -n mxnet27 python=2.7
conda info --envs
source activate mxnet27
conda list
pip install mxnet==0.12.1
But when I run C++ example files cpp-package/example/lenet.cpp I get the this segfault:
Segmentation fault: 11
This is the place in the code where the segfault is thrown:
Symbol conv1 =
Convolution("conv1", data, conv1_w, conv1_b, Shape(5, 5), 20);
I get similar segfault for the other C++ examples.
I have built MXNet on OS X 10.13.2
I disabled as many libraries as possible, e.g. OpenCV and CUDA.
On Simon Corston-Oliver suggestion I upgraded to MXNet 1.0.0, but that version did not compile with Clang on OS X. Error message:
operator_tune.h:150:36: note: add an explicit instantiation declaration to suppress this
warning if 'mxnet::op::OperatorTuneByType<float>::tuning_mode_' is explicitly instantiated in another translation unit
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/unordered_map:601:15: error: object of type 'std::__1::pair<int,
mxnet::test::perf::TimingInstrument::Info>' cannot be assigned because its copy assignment operator is implicitly deleted

I don't know of a specific issue with v0.12 that would lead to a segfault but before we dig in, I'd recommend upgrading to v1.0 which was released 2017-12-04.
If you still encounter the same problem with 1.0 we can work to debug.

I found a solution to compiling MXNet 1.0.0 posted here by helloniklas:
https://github.com/apache/incubator-mxnet/issues/9217
It involved only using make instead of CMake.
This solution worked me and compiled the code.
C++ examples runs without the seg fault, but documentation is scarce. I only got one of the to do training.

Optimization failure in theano

I am using Fedora with the Anaconda Python environment. I have a 960m nvidia gpu, for which I have installed the required drivers and the CUDA toolkit. But when I try to run the theano tests, I end up getting the following error (in a huge error output):
EE.EEEERROR (theano.gof.opt): Optimization failure due to: constant_folding
ERROR (theano.gof.opt): node: DimShuffle{x}(TensorConstant{2})
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
I was trying to compile a simple function y, when I first saw the error. Searching for a solution led me to find that a lot of people had the same problem with the test function, but without any definite solutions. I followed the theano documentations and set the $CUDA_ROOT to my cuda root folder, but to no avail.
I'm using theano version 0.8.2 and Numpy 1.11.1, both from the conda repos. Seems like it is a GPU issue. But if it has problems, shouldn't it fallback to the CPU?
Any help would be highly appreciated. Thanks!

OpenCV 3, python 3 'Image step is wrong' error

It's possible that my problem is simply a Python 3 OpenCV bug, but I don't know. I have 32 bit Python version 3.4.3 installed in Windows 10. I have OpenCV 3.0.0 32 bit installed from this website http://www.lfd.uci.edu/~gohlke/pythonlibs/ (opencv_python‑3.0.0‑cp34‑none‑win32.whl).
I also have numpy 1.10.0b1 beta installed from that site.
I've tested out the same basic program flow below using OpenCV with Java and it works. For that reason I figure this may just be a Python bug issue. What happens is that the call to drawContours in the code below produces this error:
OpenCV Error: Image step is wrong (Step must be a multiple of esz1) in cv::setSize, file ......\opencv-3.0.0\modules\core\src\matrix.cpp, line 300
The test image I am using is 1168 x 1400 pixels.
Here is the code:
import cv2
import numpy as np
img = cv2.imread('test.jpg')
imgray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
ret, threshImg = cv2.threshold(imgray,127, 255,cv2.THRESH_BINARY)
can = cv2.Canny(threshImg,100,200)
contImg, contours, hierarchy = cv2.findContours(can,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)
cv2.drawContours(img, contours,-1,(0,255,0))
cv2.imwrite('test write.jpg', img)
*******EDIT********
**I just solved the problem by installing numpy version 1.9.2 instead of the 1.10 beta.****

This has to do with development and beta releases of NumPy using relaxed strides. This is done to force detection of subtle bugs in third party libraries that make unnecessary assumptions about the strides of arrays.
Thanks to that the issue was detected a while back and is now fixed in the development version of OpenCV, see the relevant PR, but it will likely take some time until it makes it to a proper OpenCV release.
Regardless of that being fixed, as soon as the final version of NumPy 1.10 is released, you should be able to safely switch to it, even with the buggy current OpenCV version, as relaxed strides will be deactivated.

I solved the issue by installing numpy 1.9.2 instead of the new 1.10 beta version.