every time i run pyspark i got these errors and if i ignored them when i just write sc it gives NameError: name 'sc' is not defined any help ??
pyspark
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
17/08/07 13:57:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Traceback (most recent call last):
File "/usr/local/spark/python/pyspark/shell.py", line 45, in <module>
spark = SparkSession.builder\
File "/usr/local/spark/python/pyspark/sql/session.py", line 169, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/usr/local/spark/python/pyspark/context.py", line 334, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/usr/local/spark/python/pyspark/context.py", line 118, in __init__
conf, jsc, profiler_cls)
File "/usr/local/spark/python/pyspark/context.py", line 186, in _do_init
self._accumulatorServer = accumulators._start_update_server()
File "/usr/local/spark/python/pyspark/accumulators.py", line 259, in _start_update_server
server = AccumulatorServer(("localhost", 0), _UpdateRequestHandler)
File "/usr/lib/python2.7/SocketServer.py", line 417, in __init__
self.server_bind()
File "/usr/lib/python2.7/SocketServer.py", line 431, in server_bind
self.socket.bind(self.server_address)
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.gaierror: [Errno -2] Name or service not known
After 1 week of search i just found the solution just by add localhost to file /etc/hosts and then every thing went well
Related
I am using pre-built 'spark-2.0.1-bin-hadoop2.7’ and when I try to start pyspark, I get following message.
Any ideas what could be wrong? I tried using python3, setting SPARK_LOCAL_IP to 127.0.0.1 but same error.
~ -> cd /Applications/spark-2.0.1-bin-hadoop2.7/bin/
/Applications/spark-2.0.1-bin-hadoop2.7/bin -> pyspark
Python 2.7.12 (default, Oct 11 2016, 05:24:00)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
16/12/19 14:50:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/12/19 14:50:47 WARN Utils: Your hostname, XXXXXX.com resolves to a loopback address: 127.0.0.1; using XX.XX.XX.XXX instead (on interface en0)
16/12/19 14:50:47 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Traceback (most recent call last):
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/shell.py", line 43, in <module>
spark = SparkSession.builder\
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/sql/session.py", line 169, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/context.py", line 294, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/context.py", line 115, in __init__
conf, jsc, profiler_cls)
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/context.py", line 174, in _do_init
self._accumulatorServer = accumulators._start_update_server()
File "/Applications/spark-2.0.1-bin-hadoop2.7/python/pyspark/accumulators.py", line 259, in _start_update_server
server = AccumulatorServer(("localhost", 0), _UpdateRequestHandler)
File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 417, in __init__
self.server_bind()
File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 431, in server_bind
self.socket.bind(self.server_address)
File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
Thanks
Found it. Some how my host mapping was messing it up. Changing it to point to localhost worked.:
/etc/host
#127.0.0.1 XXXXXX.com
127.0.0.1 localhost
In cases when you cannot cleanup /etc/hosts (such as it's being tempered with by some VPN solution), here is a workaround:
from pyspark.sql import SparkSession
def patch_pyspark_accumulators():
from inspect import getsource
import pyspark.accumulators as pa
exec(getsource(pa._start_update_server).replace("localhost", "127.0.0.1"), pa.__dict__)
patch_pyspark_accumulators()
spark = SparkSession.builder.getOrCreate()
I'm having problems using NanoMsg. I was using default sockets before but then someone pointed out that this was a bad idea. So I searched to get a better package and found NanoMsg. When I tried importing I got this error:
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
================= RESTART: C:\Users\user\Desktop\Website.py =================
=============================== RESTART: Shell ===============================
>>> import nanomsg
Warning (from warnings module):
File "C:\Users\user\lib\site-packages\nanomsg_wrappers\__init__.py", line 22
"%s, performance may be affected!") % (default,))
UserWarning: Could not load the default wrapper for your platform: cpy, performance may be affected!
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
import nanomsg
File "C:\Users\user\lib\site-packages\nanomsg\__init__.py", line 7, in <module>
from . import wrapper
File "C:\Users\user\lib\site-packages\nanomsg\wrapper.py", line 4, in <module>
_wrapper = _load_wrapper()
File "C:\Users\user\lib\site-packages\nanomsg_wrappers\__init__.py", line 23, in load_wrapper
return importlib.import_module('_nanomsg_ctypes')
File "C:\Users\user\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "C:\Users\user\lib\site-packages\_nanomsg_ctypes\__init__.py", line 10, in <module>
_lib = ctypes.windll.nanomsg
File "C:\Users\user\lib\ctypes\__init__.py", line 417, in __getattr__
dll = self._dlltype(name)
File "C:\Users\user\lib\ctypes\__init__.py", line 347, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found
>>>
Any solutions?
I have read that the Raspberry Pi is capable of transmitting FM.
I have downloaded the script and put on the directory "/home/pi/".
Then I have executed following commands in the terminal over putty:
pi#raspberrypi ~ $ sudo python
Python 2.7.3 (default, Mar 18 2014, 05:13:23)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import PiFm
>>> PiFm.play_sound("left_right.wav")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "PiFm.py", line 7, in play_sound
call(["./pifm", filename])
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
I have ensured that the soundfile "left_right.wav" is in the same directory.
I don't know why this error is throwed.
I use Raspbian as OS, if this is revelant...
EDIT: The files used are in this download: http://cdn.makezine.com/make/pifm/PiRadio.zip
I'm trying to install pyquery on Windows and I get the following error when I try to do selects like this d('p:first'). Everything else seems to be working. Any idea what am I missing? This issue happens only on my windows machine, on my MAC works fine.
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyquery import PyQuery as pq
>>> d = pq("<p></p>")
>>> d('p:first')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\pyquery-1.2.4-py2.7.egg\pyquery\pyquery.py
", line 237, in __call__
result = self.__class__(*args, parent=self, **kwargs)
File "C:\Python27\lib\site-packages\pyquery-1.2.4-py2.7.egg\pyquery\pyquery.py
", line 213, in __init__
xpath = self._css_to_xpath(selector)
File "C:\Python27\lib\site-packages\pyquery-1.2.4-py2.7.egg\pyquery\pyquery.py
", line 223, in _css_to_xpath
return self._translator.css_to_xpath(selector, prefix)
File "build\bdist.win32\egg\cssselect\xpath.py", line 188, in css_to_xpath
File "build\bdist.win32\egg\cssselect\xpath.py", line 188, in <genexpr>
File "build\bdist.win32\egg\cssselect\xpath.py", line 208, in selector_to_xpat
h
File "build\bdist.win32\egg\cssselect\xpath.py", line 230, in xpath
File "build\bdist.win32\egg\cssselect\xpath.py", line 272, in xpath_pseudo
File "C:\Python27\lib\site-packages\pyquery-1.2.4-py2.7.egg\pyquery\cssselectp
atch.py", line 19, in xpath_first_pseudo
xpath.add_post_condition('position() = 1')
AttributeError: 'XPathExpr' object has no attribute 'add_post_condition'
>>>
I had installed pyquery with "easy_install pyquery" and this was causing this error.
I removed it and then installed it from https://github.com/gawel/pyquery with python setup.py install and now it's working.
When I run the following command, it produces the following error:
% python manage.py celeryd -l INFO ~/Workspace/django-projects/project/src
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/home/damon/Workspace/django-projects/project/env/lib/python2.6/site-packages/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/home/damon/Workspace/django-projects/project/env/lib/python2.6/site-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/damon/Workspace/django-projects/project/env/lib/python2.6/site-packages/django/core/management/__init__.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/damon/Workspace/django-projects/project/env/lib/python2.6/site-packages/django/core/management/__init__.py", line 67, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/home/damon/Workspace/django-projects/project/env/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/home/damon/Workspace/django-projects/project/env/lib/python2.6/site-packages/djcelery/management/commands/celeryd.py", line 13, in <module>
class Command(CeleryCommand):
File "/home/damon/Workspace/django-projects/project/env/lib/python2.6/site-packages/djcelery/management/commands/celeryd.py", line 17, in Command
option_list = CeleryCommand.option_list + worker.get_options()
File "/home/damon/Workspace/django-projects/project/env/lib/python2.6/site-packages/celery/bin/celeryd.py", line 89, in get_options
conf = self.app.conf
AttributeError: 'NoneType' object has no attribute 'conf'
Here's the version information:
% python ~/Workspace/django-projects/project/src
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import celery
>>> print celery.VERSION
(2, 2, 0, 'a1')
>>> import djcelery
>>> print djcelery.VERSION
(2, 2, 0, 'a1')
Should be easy to fix: just upgrade your install of django-celery.