Develop Reference

ImportError: No module named queue - Flask app on Cloud Foundry

I am trying to push a flask app (Python 3.5) to Cloud Foundry (CF). The application takes a POST request (text file) and returns a message. It works locally (tested via Postman). However, when attempting to push it to CF, it gives the error -
ImportError: No module named queue
Here is my code which contains queue.
import queue as Queue
self._batch_queue = Queue.Queue(self.BATCH_QUEUE_MAX)
self._example_queue = Queue.Queue(self.BATCH_QUEUE_MAX * self._hps.batch_size)
I've tried the solutions suggested here, but neither of these solve my problem. I think the issue is with the Python in CF not having queue package. (I could be wrong).
Anyone ideas on how to go about solving this will be very appreciated. Thanks in advance!

As mentioned in the comments by #KlausD, it seems like you have the wrong version of Python installed. On Cloud Foundry, you would set the version by including a file called runtime.txt in the root of your project (i.e. the directory from which you're running cf push).
https://docs.cloudfoundry.org/buildpacks/python/index.html#runtime
That file is used to tell the Python buildpack which version of Python to install for you. Suggestions would be python-3.5.x or python-3.6.x which would install the latest 3.5 or 3.6 release. You can specify an exact version like python-3.5.5 but it's not recommended as it's easy to forget to update that file when new versions of Python come out.
You can see which versions of Python are supported by the buildpack here.
https://buildpacks.cloudfoundry.org/#/buildpacks/python/v1.6.17
(Note that link goes to the latest version of the buildpack at the time I wrote this, it will get out of date. In the future, just click the most recent version of the buildpack to see what ships with it).

GCloud: can't import datastore

Well I'm trying to use Datastore in a personal project using the Google App Engine. Though, I can't import the datastore module, no matter how hard I try.
I've been using the online console during the whole time (in order to avoid to have to solve problems first on my PC and then on GCloud...)
So, I'm using
from google.cloud import datastore
Unfortunately, that's not working at all. The last error I have is
ImportError: No module named google.protobuf
But before I had things like Can't import Datastore.
What I did was removing the integrality of /lib, and reinstalling every dependancy with pip. Here is my requirements.txt:
# This requirements file lists all third-party dependencies for this project.
#
# Run 'pip install -r requirements.txt -t lib/' to install these dependencies
# in `lib/` subdirectory.
#
# Note: The `lib` directory is added to `sys.path` by `appengine_config.py`.
Flask==0.10
google.cloud==0.25.0
protobuf==3.3.0
(The last line was added to try to resolve the last error I got). Before having this error, I got
Also, a little clarification question: I've seen (while looking for answers) people using gcloud and some using google.cloud. What's the difference? What should I use?
Also, pip show google.cloud shows nothing.
What am I missing?
Thank you

Well, if anyone is wondering, here's how I solved the problem. What fixed it was changing the Flask version to 0.12 (don't know why, but that's what happened).
I deleted lib to be sure I was starting from scratch. Then, I used this requirements.txt file:
Flask==0.12
google-cloud==0.25.0
click==5.1
(click is needed by Flask 0.12).

Building webrtc native on Ubuntu

I am having a hard time building webrtc-native on ubuntu. The Build instructions for webrtc are wayward and poor. I am following https://webrtc.org/native-code/development/.
Figured that I need to install the depot tools and run fetch v8. Then I did gclient sync. Did apt-get install ninja. Now I am stuck at python webrtc/build/gyp_webrtc. I get the following error
Traceback (most recent call last): File "webrtc/build/gyp_webrtc",
line 23, in
import gyp_chromium ImportError: No module named gyp_chromium
I badly need help here. Also, it seems that the community doesn't care much about helping to build native apps using webrtc. The focus is on upstream/Web APIs.

Please check https://github.com/HefnySco/node-webrtc-native
I hade very hard time. I used build-webrtc with some tweaks all listed on the above link.
The linked project goes one step forward and build a node component after building the webrtc lib.

WebRTC recently moved from gyp to gn as the default method to generate ninja files.
You should find the process to get started easier here: https://webrtc.org/native-code/development/

How to run WordCountTopology from storm-starter in Intellij

I work with Storm for a while already, but want to get started with development. As suggested, I am using IntelliJ (up to now, I was using Eclipse and did only write topologies against Java API).
I was also looking at
https://github.com/apache/storm/tree/master/examples/storm-starter#intellij-idea
This documentation is not complete. I was not able to run anything in Intellij first. I could figure out, that I need to remove the scope of storm-core dependency (in storm-starter pom.xml). (found here: storm-starter with intellij idea,maven project could not find class)
After that I was able to build the project. I can also run ExclamationTopology with no problems within IntelliJ. However, WordCountTopology fails.
First I got the following error:
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.
Serializer Exception:
Traceback (most recent call last):
File "splitsentence.py", line 16, in
import storm
ImportError: No module named storm
Update: installing python-storm is not required to make it work
I was able to resolve it via: apt-get install python-storm (from StackOverflow)
However, I don't speak Python and was wondering what the problem is and why I could resolve it like this. Just want to get deeper into it. Maybe someone can explain.
Unfortunately, I am getting a different error now:
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.
Serializer Exception:
Traceback (most recent call last):
File "splitsentence.py", line 18, in
class SplitSentenceBolt(storm.BasicBolt):
AttributeError: 'module' object has no attribute 'BasicBolt'
I did not find any solution on the Internet. Asking at dev#storm.apache.org did not help either. I go the following suggestion:
I think that it was always assumed that topology would always be invoked through storm-command line. Thus working directory would be ${STORM-INSTALLATION}/bin/storm Since storm.py is in the this directory, splitSentence.py would be able to find storm modules. Can you set the working directory to a path, where storm.py is present and then try. If it works, we can add it later to the documentation
However, chancing the working directory did not solve the problem.
And as I am not familiar with Python and as I am new to IntelliJ, I am stuck now. Because ExclamationTopology runs, I guess my basic setup is correct.
What do I do wrong? It is possible at all to run WordcountTopology in LocalCluster in IntelliJ?

Unfortunately, AFAIK you can't run multilang feature with LocalCluster without having packaged file.
ShellProcess relies on codeDir of TopologyContext, which is used by supervisor.
Workers are serialized to stormcode.ser, but multilang files should extracted to outside of serialized file so that python/ruby/node/etc can load it.
Accomplishing this with distribute mode is easy because there's always user submitted jar, and supervisor can know it is what user submitted.
But accomplishing this with local mode is not easy cause supervisor cannot know user submitted jar, and users can run topology to local mode without packaging.
So, Supervisor in local mode finds resource directory ("resources") from each jars (which ends with "jar") in classpath, and copy first occurrence to codeDir.
storm jar places user topology jar to the first of classpath, so it can be run without issue.
So normally, it's natural for ShellProcess to not find "splitsentence.py". Maybe your working directory or PYTHONPATH did the trick.

I struggled with a similar issue, not with the sample topology, but with my own using a Python bolt.
Also experienced the "AttributeError: 'module' object has no attribute 'BasicBolt'" exception - in local mode and when submitting to the cluster.
There are very few resources on this, I found your question and little else discussing this issue.
In case someone else has the same problem:
Make sure you include the correct Maven "multilang-python" dependency in your pom file. This will package the correct run time dependencies into the JAR file needed to run your topology.

I managed to run it on my virtualbox, storm version 1.2.2:
just download https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py and put it into any folder you want, for example: /apache-storm-1.2.2/examples/storm-starter/multilang/resources/ , and then change the main function:
public static void main(String[] args) throws Exception {
SplitSentence pythonSplit = new SplitSentence();
Map env = new HashMap();
env.put("PYTHONPATH", "/apache-storm-1.2.2/examples/storm-starter/multilang/resources/");
pythonSplit.setEnv(env);
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomSentenceSpout(), 5);
builder.setBolt("split",pythonSplit, 8).shuffleGrouping("spout");
builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word"));
Config conf = new Config();
conf.setDebug(true);
if (args != null && args.length > 0) {
conf.setNumWorkers(3);
StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology());
}
else {
conf.setMaxTaskParallelism(3);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("word-count", conf, builder.createTopology());
Thread.sleep(600000);
cluster.shutdown();
}
}
the full instructions can be found on my blog which includes other issues encountered when running it in Local Mode and Local Cluster mode: https://lyhistory.com/storm/

Python Pylearn2 package "ImportError: No module named pylearn2.utils"

I have recently tried to use pylearn2, a deep machin learning package for Python developed at University of Montreal.
I've just installed it and tried to run a simple example, but it did not work.
I have been using a pc with an Ubuntu 13.10 system, on which I found ipython installed.
I have installed Theano and later pylearn2, by following this webpage instructions:
http://deeplearning.net/software/pylearn2/
I have also modified the .bashrc file, as suggested
I thought that everything went well, and then I tried this Quick start example:
http://deeplearning.net/software/pylearn2/tutorial/index.html
I stopped at the first command:
python make_dataset.py
My terminal states:
Traceback (most recent call last): File "make_dataset.py", line 14,
in
Do you have any ideas on why it is not working?
Do you why these errors occur?
Thanks a lot
EDIT: the 14 line is the first non-commented line of the file. It states
from pylearn2.utils import serial

Without more information, I can only guess, but my first guess is…
You haven't actually installed pylearn2, because if you follow the linked docs to grab the git repo and add a PYLEARN2_DATA_PATH variable, nothing gets installed into site-packages (or dist-packages or anywhere else on sys.path).
This means that pylearn2 will only work when you start Python from within the top-level directory of the pylearn2 repo.
So, if you run a script like this:
$ cd /path/to/pylearn2
$ cd scripts/tutorials/grbm_smd/
$ python make_dataset.py
… it won't actually work.
It looks like there is a setup.py file in the repository. Does it work? I have no idea. Even though the docs don't mention using it, you might want to try. Either this:
$ pip install .
… or, if you don't have pip or it doesn't work on this package:
$ python setup.py install
Either way, of course, you may need sudo or a flag to install to your user site-packages instead of system, etc., as with any other Python package.
If that doesn't work, you might be able to just add /path/to/pylearn2 to your sys.path in some way. The most obvious way is by doing an export PYTHONPATH=/path/to/pylearn2:$PYTHONPATH in your ~/.bashrc.
Also, you will need to either source ~/.bashrc or create a new shell to get any effects of modifying the file.
If you're wondering why the instructions and the tutorial together don't give you enough information to make this work without a lot of hassle, I think that's covered in the very top of the documentation:
Pylearn2 is still undergoing rapid development. Don’t expect a clean road without bumps!
And the very fact that there is no PyPI download yet implies that this really is not ready for novices to use. If you don't know enough about using Python packages (and bash basics) to muddle through on your own, there's a good chance you won't be able to use this package.