I work with Storm for a while already, but want to get started with development. As suggested, I am using IntelliJ (up to now, I was using Eclipse and did only write topologies against Java API).
I was also looking at
https://github.com/apache/storm/tree/master/examples/storm-starter#intellij-idea
This documentation is not complete. I was not able to run anything in Intellij first. I could figure out, that I need to remove the scope of storm-core dependency (in storm-starter pom.xml). (found here: storm-starter with intellij idea,maven project could not find class)
After that I was able to build the project. I can also run ExclamationTopology with no problems within IntelliJ. However, WordCountTopology fails.
First I got the following error:
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.
Serializer Exception:
Traceback (most recent call last):
File "splitsentence.py", line 16, in
import storm
ImportError: No module named storm
Update: installing python-storm is not required to make it work
I was able to resolve it via: apt-get install python-storm (from StackOverflow)
However, I don't speak Python and was wondering what the problem is and why I could resolve it like this. Just want to get deeper into it. Maybe someone can explain.
Unfortunately, I am getting a different error now:
java.lang.RuntimeException: backtype.storm.multilang.NoOutputException: Pipe to subprocess seems to be broken! No output read.
Serializer Exception:
Traceback (most recent call last):
File "splitsentence.py", line 18, in
class SplitSentenceBolt(storm.BasicBolt):
AttributeError: 'module' object has no attribute 'BasicBolt'
I did not find any solution on the Internet. Asking at dev#storm.apache.org did not help either. I go the following suggestion:
I think that it was always assumed that topology would always be invoked through storm-command line. Thus working directory would be ${STORM-INSTALLATION}/bin/storm Since storm.py is in the this directory, splitSentence.py would be able to find storm modules. Can you set the working directory to a path, where storm.py is present and then try. If it works, we can add it later to the documentation
However, chancing the working directory did not solve the problem.
And as I am not familiar with Python and as I am new to IntelliJ, I am stuck now. Because ExclamationTopology runs, I guess my basic setup is correct.
What do I do wrong? It is possible at all to run WordcountTopology in LocalCluster in IntelliJ?
Unfortunately, AFAIK you can't run multilang feature with LocalCluster without having packaged file.
ShellProcess relies on codeDir of TopologyContext, which is used by supervisor.
Workers are serialized to stormcode.ser, but multilang files should extracted to outside of serialized file so that python/ruby/node/etc can load it.
Accomplishing this with distribute mode is easy because there's always user submitted jar, and supervisor can know it is what user submitted.
But accomplishing this with local mode is not easy cause supervisor cannot know user submitted jar, and users can run topology to local mode without packaging.
So, Supervisor in local mode finds resource directory ("resources") from each jars (which ends with "jar") in classpath, and copy first occurrence to codeDir.
storm jar places user topology jar to the first of classpath, so it can be run without issue.
So normally, it's natural for ShellProcess to not find "splitsentence.py". Maybe your working directory or PYTHONPATH did the trick.
I struggled with a similar issue, not with the sample topology, but with my own using a Python bolt.
Also experienced the "AttributeError: 'module' object has no attribute 'BasicBolt'" exception - in local mode and when submitting to the cluster.
There are very few resources on this, I found your question and little else discussing this issue.
In case someone else has the same problem:
Make sure you include the correct Maven "multilang-python" dependency in your pom file. This will package the correct run time dependencies into the JAR file needed to run your topology.
I managed to run it on my virtualbox, storm version 1.2.2:
just download https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py and put it into any folder you want, for example: /apache-storm-1.2.2/examples/storm-starter/multilang/resources/ , and then change the main function:
public static void main(String[] args) throws Exception {
SplitSentence pythonSplit = new SplitSentence();
Map env = new HashMap();
env.put("PYTHONPATH", "/apache-storm-1.2.2/examples/storm-starter/multilang/resources/");
pythonSplit.setEnv(env);
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomSentenceSpout(), 5);
builder.setBolt("split",pythonSplit, 8).shuffleGrouping("spout");
builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word"));
Config conf = new Config();
conf.setDebug(true);
if (args != null && args.length > 0) {
conf.setNumWorkers(3);
StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology());
}
else {
conf.setMaxTaskParallelism(3);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("word-count", conf, builder.createTopology());
Thread.sleep(600000);
cluster.shutdown();
}
}
the full instructions can be found on my blog which includes other issues encountered when running it in Local Mode and Local Cluster mode: https://lyhistory.com/storm/
Related
I want to run JdeRobot drone_cat_mouse on my Ubuntu 20.04. I'm using ROS Noetic and has faithfully followed these installation instructions. Everything it told me to test was working properly.
When I first ran roslaunch drone_cat_mouse.launch, there was an import error for teleopWidget and sensorsWidget which I fixed by using relative imports. Then I had an error No module named qfi.
Unlike teleopWidget and sensorsWidget, I couldn't find the qfi module in the JdeRobot/drones source code. So I googled it, and the only relevant result that popped up was this, which led to this link. They said to:
sudo touch /usr/lib/python2.7/dist-packages/qfi/__init__.py
But I ran that command and this happened!
Not even pip has a "qfi" module!
So I thought to check JdeRobot's entire repository. Turns out it was in JdeRobot/base, and that repo is not maintained anymore!
After further digging, there was this issue which basically tells us forget about it and move to the web release! But I can't, circumstances forced me to use the source code option (deliverables are drone_cat_mouse.world and my_solution.py, it's impossible for me to get the former in the docker web version and the latter's format is different between the source code version and the web version).
In a nutshell, how do I fix this qfi module problem so that I can run the exercises from source like these people?
I'm just stupid, as usual. all I need to do was clone https://github.com/JdeRobot/ThirdParty, get the qfi module, copy it to
~/catkin_ws/src/drones/rqt_drone_teleop/src/rqt_vel_teleop/ and replace all qfi imports with its relative import version. All common sense
No errors in terminal, gazebo runs, but somehow the rqt widget for drone vision never appears.
Forget it, I'm giving up on this dumpster fire of a program.
Edit: I did another fresh install, followed the steps, noticed troubleshooting for qfi which required qmake, but same end result
If you're trying to launch drone_cat_mouse there is an issue with the namespace of the RQT widget that occurs when you try to launch it.
Namely, the topics that exist for drone_cat_mouse are prefixed by cat/ or mouse/. But RQT will try to access these topics without the prefix and run into an error. Alternatively, since you have a local install, you can try to run the code manually by running
python my_solution.py
Just make sure the change the area where the DroneWrapper class was called in the following manner:
HAL = DroneWrapper('drone', 'cat/')
Here 'drone' is the name of the node you are creating and 'cat/' is the namespace given to the DroneWrapper Class.
I have a Windows machine in which some Python code works, but now needs to work on new machine.
I have installed the same Python version 3.6.5.
First issue was that when I run the code, it can not find a re.py library, which is in fact in Python's Lib folder. So I have added sys.path.append('C:\Python\Lib') and now it can find it.
But now I get the syntax error from that library, where I say import re, that lines throws an error regarding some line in re.py library. If I import getopt, I also get syntax error on some line.
How is that even possible? Syntax error in pythons Lib files which came with installation?
And the thing is that on machine 1 it works, same file contents, same python version. I am under impression that I have wrong in python.exe version for this version of libraries, but I have simply downloaded Windows installer and installed it.
I don't even know what to google for, does someone has any idea? I am importing re in WeblogicAuto.py on line 5.
D:\Jenkins\workspace\weblogic-full-deployment-copy\weblogic-deployment>MainAutoDeployment.py -f DEV -v 2.61.0.12
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Problem invoking WLST - Traceback (innermost last):
File "D:\Jenkins\workspace\weblogic-full-deployment-copy\weblogic-deployment\WeblogicAuto.py", line 5, in ?
File "C:\Python\Lib\re.py", line 247
b"_abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890")
^
SyntaxError: invalid syntax
You are using WLST (WebLogic Scripting Tool).
WLST is very specific distribution of python:
it's jython
it roughly corresponds with python 2.7
And you trying to import libs from your CPython 3.6.5 distribution.
You need to:
remove your sys.path.append that you added earlier
check python libraries in jython library path. Which could be $HOME/.jython or WLST-specific path (look in WebLogic documentation)
OK many thanks to ya all, I've found the solution, it was about Jython libraries. I will try to elaborate a bit if someone stumbles upon this issue.
What I do here is that I deploy java modules to Oracle Weblogic application server. So this answer will also help someone trying to deploy to Weblogic from python.
From the start I was avoiding to install Webloglogic software on a machine from where I do the deploy (and that is a Jenkins slave which runs these python file, simple job).
I noticed that first machine (first Jenkins slave) has Weblogic installed, but I like to keep it minimal :)
What you do need is weblogic.jar and (not sure at this moment) wlfullclient.jar which is generated on Weblogic server (google how if needed or you may already have it).
The thing is that even though I have pure python code, when you call another python code with java weblogic.WLST pythonCode.py, it will be run with jython application! And it needs its libraries in its sys path.
In jython file I've added print(sys.path) then run in on both machines (slaves). I've noticed that path is different on those machines, even though if you type it in command prompt you get the same, but different then when Jenkins runs it.
So instead of looking how to fix those paths and copy files to them, I have created folders where it expects them, and copied them from first machine (easy fix, I may look into it later).
These are the sys.path and files that were needed, present on first machine:
['D:\\Jenkins\\weblogic\\Lib', '__classpath__', 'C:/bea10/wlserver_10.3/server/lib/weblogic.jar', 'C:/bea10/wlserver_10.3/common/wlst/modules/jython-modules.jar/Lib', 'C:/bea10/wlserver_10.3/common/wlst', 'C:/bea10/wlserver_10.3/common/wlst/lib', 'C:/bea10/wlserver_10.3/common/wlst/modules', '.']
This was sys.path on second machine, so I simply copied there:
['D:\\Jenkins\\weblogic\\Lib', '__classpath__', 'D:/Jenkins/server/lib/weblogic.jar', 'D:/Jenkins/common/wlst/modules/jython-modules.jar/Lib', 'D:/Jenkins/common/wlst', 'D:/Jenkins/common/wlst/lib', 'D:/Jenkins/common/wlst/modules', '.']
Note that jython-modules.jar is a file, so /Lib should be from that file if I get how java works.
Feel free to contact me for more details.
Whenever I start a python script that tries to access the keyboard, for example, with OpenCV's waitKey() method, I get the following error:
Qt: Failed to create XKB context!
Use QT_XKB_CONFIG_ROOT environmental variable to provide an additional search path, add ':' as separator to provide several search paths and/or make sure that XKB configuration data directory contains recent enough contents, to update please see http://cgit.freedesktop.org/xkeyboard-config/ .
This error started to appear when I installed the keyboard library (or pynput, I am not quite sure) and persisted after I uninstalled them. I do not have QT installed.
Also, neither OpenCV's waitKey nor the two installed libraries can detect any key events (I suspect this is connected), which was already the case before I got the error though.
I am running on Ubuntu 18.04.
How can I go about this problem?
I think this recent issue was resolved here:
https://github.com/conda-forge/qt-feedstock/issues/104
I think the solution is
export QT_XKB_CONFIG_ROOT=/usr/share/X11/xkb
but I'm not an expert... somebody back me up here please.
I am following the tutorial here:
http://www.prokopyshen.com/create-custom-zipline-data-bundle
and trying to set up a custom bundle to get price from custom, non US financial assets. I am stuck on the line that says:
Advise zipline of our bundle by registering it via .zipline/extension.py
My extension.py file is located in the .zipline/ directiory and has the following code:
from zipline.data.bundles import register
from zipline.data.bundles.viacsv import viacsv
eqSym = {
"CBA"
}
register(
'CBA.csv', # name this whatever you like
viacsv(eqSym),
)
I don't get what it means to register the bundle via .zipline/extension.py though? I thought it might mean to just run the extension.py file from my terminal via a:
python extenion.py
but that fails and says:
ImportError: No module named viacsv
How do i register this bundle?
I also followed this tutorial and I must confess this part is a little confusing.
First of all, I don't think it's necessary to run:
$ python extension.py
The error message you get probably comes from the fact that Python cannot find the viacsv.py file in sys.path (the places where it looks for modules, etc.). In the tutorial you mentioned, it's not really clear what to do with this file. As far as I am concerned, I just saved the viacsv.py file in my local site-packages directory. As I am on Linux I put it there ~/.local/lib/python2.7/site-packages but it might different for you. You can run the following python script to find out:
import sys
for dr in sys.path:
print dr
Then I just substituted from zipline.data.bundles.viacsv import viacsv with from viacsv import viacsv in extension.py.
I suspect you might be looking for the wrong place for the extension.py file.
For windows machine, the file is under "~\.zipline\extension.py". In my case, it's under "C:\Users\XXXX\.zipline\extension.py".
I had been looking at zipline folder under conda's site-packages folder, and couldn't find it. Then created an extension.py myself wondering why it's not called.
Check a related post here https://www.quantopian.com/posts/zipline-issue-while-creating-custom-bundle-to-bring-yahoo-data.
Same issue here, #Gillu13 pointed me to this solution.
I installed zipline through conda. So zipline is installed in
home/me/anaconda3/envs/krakenex/lib/python3.6/site-packages
in there you will find zipline/data/bundles and you can put viacsv.py in there...
then
from zipline.data.bundles.viacsv import viacsv
works
For three days, I have been pulling my hair out trying to wrap my head around Okta & SAML.
On my local machine (OSX Mavericks), I am able to successfully follow the steps listed here: http://developer.okta.com/docs/guides/pysaml2
Things work.
But moving everything over to our production server, which is a CentOS box, running nearly identical code, I am faced with this "SignatureError: Failed to verify signature" error.
Traceback (most recent call last):
auth_response = saml_client.parse_authn_request_response(SAMLResponse, entity.BINDING_HTTP_POST)
File \"/usr/local/lib/python2.7.11/lib/python2.7/site-packages/saml2/client_base.py\", line 599, in parse_authn_request_response binding, **kwargs)
response = response.loads(xmlstr, False, origxml=origxml)
File \"/usr/local/lib/python2.7.11/lib/python2.7/site-packages/saml2/response.py\", line 510, in loads self._loads(xmldata, decode, origxml)
File \"/usr/local/lib/python2.7.11/lib/python2.7/site-packages/saml2/response.py\", line 335, in _loads **args)
File \"/usr/local/lib/python2.7.11/lib/python2.7/site-packages/saml2/sigver.py\", line 1756, in correctly_signed_response
class_name(response), origdoc)
File \"/usr/local/lib/python2.7.11/lib/python2.7/site-packages/saml2/sigver.py\", line 1571, in _check_signature
raise SignatureError(\"Failed to verify signature\")
SignatureError: Failed to verify signature
I have scoured the internet looking for a way to troubleshoot this error. I am new to SAML and Okta.
My assumption is that this has something to do with xmlsec1 acting differently on our production machine. But the versions are identical. There are many dependencies so I'm not sure where the problem might be.
Has anyone ran into this error? Any thoughts on what I might be able to try?
I know this is a little late, but in case someone else runs into this:
pysaml2 provides a lot of logging using python's built in logging, I defined a handler for saml2.sigver and that gave a lot of info. In those logs I found this:
Error: unable to load xmlsec-openssl library. Make sure that you have
this it installed, check shared libraries path (LD_LIBRARY_PATH)
envornment variable or use "--crypto" option to specify different
crypto engine.
Turns out I needed to install xmlsec1-openssl.
Hope this helps someone in the future.
Dealing with xmlsec1 can be extremely frustrating!
The main thing that I suggest doing is enabling debugging in PySAML2, and/or setting the PYSAML2_KEEP_XMLSEC_TMP environment variable, and/or manually enable this code path in sigver.py - the general idea is to get a look at xmlsec1 command that PySAML2 is calling and have PySAML2 leave the temporary files around so that you can test the commands yourself.
As I recall, most of the issues that I've run into in the past involved PySAML2 not finding the xmlsec1 binary. The get_xmlsec_binary() function in sigver.py is responsible for finding the xmlsec1 binary. I suggest that you take a look at the code in get_xmlsec_binary() and make sure that it is looking in the right places on your system.
Depending on the operational system you will also need to install additional libraries.
In my case, I got the issue on a CentOS server, so I needed to install more 2 dependencies in addition to xmlsec1:
yum install libffi-devel xmlsec1 xmlsec1-openssl
This solved my problem.
You should also have a look on Okta's documentation. They have a guide on how to use PySAML2 to add support for Okta (via SAML) to applications written in Python.
https://developer.okta.com/code/python/pysaml2/