Develop Reference

Python 3.10 is the default PyCharm update for pyspark/spark 3.0.0.preview - are these two compatible?

There is documentation that indicates Python 3.9 is the last version compatible with Spark 3.0.0.preview. For someone new at setting up compatibility on Windows using Hadoop, Spark, Scala, Python, and PyCharm, the number of possibilities for compatibility is daunting and while browsing through the first few, given the course suggestions, it is trying to find python and spark compatibility. The version of interpreter used for the Anaconda3 is 3.8.8.
In the image below, it would appear from the school of hard knocks, the compatibility may not exist. The downloads are not indicative of versioning for Python. The Scala with the course is 2.11. The latest release also indicates 2.12 is needed for Scala (same link). Somewhere within all these choices (Using Windows 10), there is a solution but it is elusive. Hadoop version seems to be an issue as well. (Hadoop 2.7)
Other compatibility issues occur prior to setting Edit Configurations for each project and include missing files or access errors (this is a company machine - privileged management - but not full admin rights)
From the thread dump, there appears to be a log of information but being new again, it is difficult to sort through all the debug information. Possibly the JDK is wrong? but it is the one suggested to use with JRE1.8.0_201. I saw some issues as well about the possible space in "Program Files" being problematic with the JAVA_HOME path. Java did not seem happy when during one of the trial setups, it was installed to a different directory.
Anaconda3>pycharm
CompileCommand: exclude com/intellij/openapi/vfs/impl/FilePartNodeRoot.trieDescend bool exclude = true
2022-11-04 08:40:31,040 [ 1132] WARN - #c.i.o.f.i.FileTypeManagerImpl -
com.adacore.adaintellij.file.AdaSpecFileType#4f671e00 from 'PluginDescriptor(name=Ada, id=com.adacore.Ada-IntelliJ, descriptorPath=plugin.xml, path=~\AppData\Roaming\JetBrains\PyCharmCE2022.2\plugins\Ada-IntelliJ, version=0.6-dev, package=null, isBundled=false)' (class com.adacore.adaintellij.file.AdaSpecFileType) and
com.adacore.adaintellij.file.AdaBodyFileType#22a64016 from 'PluginDescriptor(name=Ada, id=com.adacore.Ada-IntelliJ, descriptorPath=plugin.xml, path=~\AppData\Roaming\JetBrains\PyCharmCE2022.2\plugins\Ada-IntelliJ, version=0.6-dev, package=null, isBundled=false)' (class com.adacore.adaintellij.file.AdaBodyFileType)
both have the same .getDisplayName(): 'Ada'. Please override either one's getDisplayName() to something unique.
com.intellij.diagnostic.PluginException:
com.adacore.adaintellij.file.AdaSpecFileType#4f671e00 from 'PluginDescriptor(name=Ada, id=com.adacore.Ada-IntelliJ, descriptorPath=plugin.xml, path=~\AppData\Roaming\JetBrains\PyCharmCE2022.2\plugins\Ada-IntelliJ, version=0.6-dev, package=null, isBundled=false)' (class com.adacore.adaintellij.file.AdaSpecFileType) and
com.adacore.adaintellij.file.AdaBodyFileType#22a64016 from 'PluginDescriptor(name=Ada, id=com.adacore.Ada-IntelliJ, descriptorPath=plugin.xml, path=~\AppData\Roaming\JetBrains\PyCharmCE2022.2\plugins\Ada-IntelliJ, version=0.6-dev, package=null, isBundled=false)' (class com.adacore.adaintellij.file.AdaBodyFileType)
both have the same .getDisplayName(): 'Ada'. Please override either one's getDisplayName() to something unique.
2022-11-04 08:40:45,822 [ 15914] SEVERE - #c.i.u.m.i.MessageBusImpl - PyCharm 2022.2.3 Build #PC-222.4345.23
2022-11-04 08:40:45,825 [ 15917] SEVERE - #c.i.u.m.i.MessageBusImpl - JDK: 17.0.4.1; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o.
2022-11-04 08:40:45,826 [ 15918] SEVERE - #c.i.u.m.i.MessageBusImpl - OS: Windows 10
[1]: https://spark.apache.org/downloads.html
[2]: https://i.stack.imgur.com/C6BGc.png

First, don't use preview releases. It's been over 2 years since Spark 3 was released; use at least the latest minor release.
That being said, Spark 3 mostly is meant to be used with Hadoop 3. All should work fine on Windows, using Java 11 (your logs say 17)
You can use Scala 2.12 or 2.13 for Spark.
Pyspark support should be fine on Python 3.9. Don't use Anaconda if you don't need it. Download Python directly. pip install pyspark. That's it. You don't even need Hadoop to run Spark code.
And unclear why you're trying to run Pycharm from the terminal. Start with spark-shell, then if that works, then you can run pyspark. Then you can use spark-submit. Only once those work, should you realistically move towards an IDE.
Alternatively, don't pollute your host machine with a bunch of software, install Docker instead, and use that to run Jupyter with Spark pre-configured - https://jupyter-docker-stacks.readthedocs.io/en/latest/using/running.html

Oracle DPI-1047 on macOS 10.14.6 with Python 2.7.10

I am quite puzzled by the following situation: a colleague and I both use Mac OS X. I use 10.15.3, python 2.7.15, GCC 4.2.1, cx_Oracle 7.0.0 with the SQL Developer 19.1.0. She uses 10.14.6, python 2.7.10, GCC 4.2.1, cx_Oracle 7.3.0 with the SQL Developer 18.3.0.
We have a simple python script that connects to a data base via cx_Oracle. We got the usual DPI-1047 error. I followed the instructions here https://oracle.github.io/odpi/doc/installation.html#macos and with the latest 19 "Basic" version, and a bit of tweaking because of the notarization (https://github.com/oracle/python-cx_Oracle/issues/341#issuecomment-541794557) I could make it work: the .dylib files are properly picked up and the code can connect to the db.
For my colleague, the file cannot be found. We followed the exact same steps as for my computer, downloaded first the 19 then the 18 "Basic" version (naively thinking it would be related to the SQL Developer version), unzipped, and added the link to the $LD_LIBRARY_PATH variable. However, the DPI-1047 error remains to pop up.
We tried to follow previous solutions in this or other forums, but they wouldn't apply or work for us:
https://github.com/oracle/python-cx_Oracle/issues/210 (with https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html#ic_x64_inst) -> the same for us except the "per step 3" part does not apply -> we do not have any .so files in the instantclient download
DPI-1047: 64-bit Oracle Client library cannot be loaded - we verified that we both use 64-bit python using python -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)', rest in this thread applies to Windows
How to fix: cx_Oracle.DatabaseError: DPI-1047: Cannot locate a 64-bit Oracle Client library - Python - there is no /usr/lib/oracle folder, neither for me nor for her, where one would need to set the version explicitly; in fact, the downloaded folder contains a soft link to the proper version of the .dylib already (we also played around with having it point to other .dylib versions)
Thank you for your help!
c.

In summary you have an issue on the 10.14.6 (Mojave) computer.
My initial thought was that you are using the OS default Python. This won't work because Apple has locked it down. From the install instructions: "Make sure you are not using the bundled Python. This has restricted entitlements and will fail to load Oracle client libraries." (I have not/cannot venture into hacking Instant Client to see if it can be made to work)
However, where did Python 2.7.10 come from? My system default is Python 2.7.16.
Don't set LD_LIBRARY_PATH. Instead my ~/lib directory has just one symbolic link:
libclntsh.dylib -> /Users/cjones/instantclient_19_3/libclntsh.dylib
All the configuration issues are due to Apple's ever tightening security requirements. Hopefully the next Oracle Instant Client release will make it easier to use.
SQL Developer is almost completely unrelated to Python cx_Oracle usage (unless you use thick-JDBC connections in SQL Developer, which is rare).
And don't follow Linux or Windows instructions on macOS!

DPI-1050: Oracle Client library is at version 0.0 but version 11.2 or higher is needed

I'm trying to connect to my work's remote DB but when I run the python script it shows this error.
I already did everything this guide says I should have as requirements but the error keeps showing. I've tried to connect through sql* and it works as it should. I downloaded oracle client libraries 12.2 (32 bits version since it needs to have the same architecture as python), I'm working with Oracle Database 11g version 11.2.0.3, Python 3.7 and I already installed Visual Studio Redistributable 2013.
This is the python script I'm using to connect to the DB. Obviously, the IP and port are different from the real one, as well as the credentials.
import cx_Oracle
host='196.0.0.0'
port='8080'
service_name='SID'
pwd= 'PASS'
dsn_tns = cx_Oracle.makedsn(host,port,service_name)
connection = cx_Oracle.connect('user',pwd,dsn_tns)
Whenever I run it, I get the error eventhough the oracle client libraries folder are already on the PATH of my computer. Do I have to configure something from the server side? Thanks for the help!

In fact, it's a version problem. I found the same problem when i used version 19. I have solved this problem when i changed the version to 12.2

I had the exact same issue and I referred to the document you linked here. I am using PyCharm as the IDE and using the python.exe which pycharm uses, I found it is 32 bit (just click the python.exe and the command prompt like screen opens and first line tells if it is 32bit or 64 bit).
Then using the below link, I downloaded the Oracle Instant Client.
https://www.oracle.com/technetwork/database/database-technologies/instant-client/downloads/index.html
I chose the 32 bit and 12.2 version (latest was v18.x at the time of writing this). This 32bit has to match your python version (which was also 32 bit in my case).
Then I extracted the zip file of instant client, put it in C:\oracle folder (doesn't matter where you put) and the my directory structure was..
C:\oracle\instantclient_12_2
Now I put this location as first location in my "Path" system environment variable of Windows machine.
After doing all this, I restarted machine (not needed) and used pycharm and connected like this...
import cx_Oracle as cxo
conn = cxo.connect("user_id", "password", "host_name:port_no/sid")
print(conn.version)
cur = conn.cursor()
cur.execute('SELECT \'X\' FROM DUAL')
for result in cur:
print(result[0])
cur.close()
conn.close()

You need to install same version of Python and database client.
For Example: 64 bit python and 64 bit client or both 32 bit will work.
I tired with Oracle XE 18c and python-3.8.3-amd64.exe

The error message has been improved in cx_Oracle 7.1 -- but the problem is that an older version (than 11.2) of the Oracle Client libraries has been detected. You need to make sure that the newer version you've installed is being detected. You can do that by ensuring that your PATH environment variable contains your 12.2 client libraries first. In the past, older versions of the Oracle Client libraries were often installed into C:\windows\system32.

The problem might be related to this issue.
In short:The Oracle Instant Client version 19.3.0.0.0 is not supported on Windows 7.
It is also mentioned in Operating System Checklist for Oracle Database Client Installation.

I had same problem DPI-1050: Oracle Client library is at version 0.0 but version 11.2 or higher is needed on Windows 10 with PyCharm and Selenium
I found that application which is installed in my pc contains oic.dll file. I know that application is in very old technology and first tried with that. Renamed that file to other and problem was solved. Connection with DB was successful
Only problem in my case is that next time when I will start that application it need that file, so i need to change file name back to oic.dll.

How to use lxml version 3.7.3 on app engine?

On their Built-in Third-party Libraries page, it says that lxml 3.7.3 is supported but I can't seem to find a way to use that.
If I write in app.yaml:
- name: lxml
version: latest
When I log etree.LXML_VERSION inside the app, I get (2, 3, 0, 0).
Setting the version specifically to "3.7.3" (instead of latest) doesn't seem to work. When I try to deploy, gcloud says:
ERROR: (gcloud.app.deploy) An error occurred while parsing file: [... /app.yaml]
lxml version "3.7.3" is not supported, use one of: "2.3", "2.3.5"
in "... app.yaml", line 17, column 1
I have installed lxml locally inside the ./lib folder, but the folder is ignored on deploy.
Am I doing something wrong?

The problems seemed to happen because of an old google cloud sdk.
The sdk was installed using apt-get and somehow it was not getting updated. This also included an older Python SDK
At this moment the latest Python SDK is 1.9.62, I was using 1.9.52.
Updating the sdk seemed to fix the problem
sudo apt-get update && sudo apt-get install google-cloud-sdk

It wouldn't be the 1st time that inconsistencies exist between the documentation, what's included in the cloud SDK, what's included in the GAE language specific SDK and what's actually available on GAE (in production). See, for example, PyCharm - Can't create App Engine Application using DJango.
The deployment error message you got suggests that the 3.7.3 version doesn't actually exist on GAE, despite being marked as available in the documentation.
The lxml library is on the list of a few special libraries which need extra attention. Most likely because they aren't "pure Python code with no C extensions" and thus they can't be included in the SDKs with the other libraries, so they need to be installed separately on your local system. From Using built-in bundled libraries with the local development server (check that entire section for related info):
Many of the built-in libraries provided by the runtime are
automatically available to the local development server. However, the
following built-in libraries must be installed locally before you can
use them with the local development server:
lxml
matplotlib
mysqldb
numpy
PIL
crcmod
pycrypto
Now, if you really want the 3.7.3 version, you may be out of luck: if indeed it's not "pure Python code with no C extensions" then you also won't be able to vendor it into your app either. Still, it's worth a try. A new try, note that you also need to:
take it out of the app.yaml file's libraries: section - you're not requesting a GAE-provided library anymore
complete the entire vendoring in procedure (you didn't mention creating the appengine_config.py file, for example).
If that doesn't work then you'll have to settle for one of the available versions mentioned in the deployment error messages. You'll need to:
add that version to the app.yaml file's libraries: section
install that version on your local system
drop the vendoring in attempt, if you went for it

PostgreSQL unable to create plpythonu extension

I'm trying to write a function in PostgreSQL on Windows with a Python script in the body and i'm running into an error message when trying to create the plpythonu extension. The command I'm running is:
CREATE EXTENSION plpythonu;
Which produces the following error message:
ERROR: could not access file "$libdir/plpython2": No such file or directory
SQL state: 58P01
I also tried running:
CREATE EXTENSION plpython3u;
which results in this error:
ERROR: could not load library "C:/Program Files (x86)/PostgreSQL/9.2/lib/plpython3.dll": The specified module could not be found.
SQL state: 58P01
The plpython3.dll file exists at this location, but apparently is missing some critical dependency. I've searched everywhere and found nothing helpful on this. I have both Python 2 and 3 installed on the machine...

The newest (9.4 or later) binary installations from EnterpriseDB contain only plpython3u.dll. In versions 9.4 to 9.6 I had to install python 3.3 to get plpython3u run.
You can check which version of Python is needed by plpython3u.dll using Dependency Walker.

A full answer can be found:
https://postgresrocks.enterprisedb.com/t5/PostgreSQL/unable-to-install-pl-python-extension/m-p/4090
It assumes you have used stackbuilder to install the edb language pack.
Do check the commands for correctness in your installation.
E.g. path to postgresql data, install path of edb and python version.
When you use depency walker (depends.exe), only pay attention to the pythonxx.dll. With older PG versions, this may or may not agree to the version installed by the EDB languages package. For version 10.7, version 3.4 Python is required. For windows, the later 3.4 Python versions do not appear to have a msi installer. You may have to install 3.4.4, or try to upgrade PG 10 to the latest version (10.11) first. This version requires python 3.7, so then you can use the EDB download.
But the python version may already exist and be found.
could not load library plpython3.dll (here on stackoverflow) was somewhat close, but did not detail the environment vars needed.
the solution proposed does not require you to change env vars permanently, which is a great help when using several python installations.