Develop Reference

Setting Connection String as App Setting/Environment Variable in Azure Function

In my Azure Function, I have specified an Environment Variable/App Setting for a database connection string. I can use the Environment Variable when I run the Function locally on my Azure Data Science Virtual Machine using VS Code and Python.
However, when I deploy the Function to Azure, I get an error: KeyValue is None, meaning that it cannot find the Environment Variable for the connection string. See error:
Exception while executing function: Functions.matchmodel Result: Failure
Exception: KeyError: 'CONNECTIONSTRINGS:PDMPDBCONNECTIONSTRING'
Stack: File "/azure-functions
host/workers/python/3.7/LINUX/X64/azure_functions_worker/dispatcher.py", line 315, in
_handle__invocation_request self.__run_sync_func, invocation_id, fi.func, args)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/azure-functions-host/workers/python/3.7/LINUX/X64/azure_functions_worker/dispatcher.py",
line 434, in __run_sync_func
return func(**params)
File "/home/site/wwwroot/matchmodel/__init__.py", line 116, in main
File "/home/site/wwwroot/matchmodel/production/dataload.py", line 28, in query_dev_database
setting = os.environ["CONNECTIONSTRINGS:PDMPDBCONNECTIONSTRING"]
File "/usr/local/lib/python3.7/os.py", line 679, in __getitem__
raise KeyError(key) from None'
I have tried the following solutions:
Added "CONNECTIONSTRINGS" to specify the Environment Variable in the Python script (which made it work locally)
setting = os.environ["CONNECTIONSTRINGS:PDMPDBCONNECTIONSTRING"]
Used logging.info(os.environ) to output my Environment Variables in the console. My connection string is listed.
Added the connection string as Application Setting in the Azure Function portal.
Added the Connection String as Connection Strings in the Azure Function portal.
Does anyone have any other solutions that I can try?

Actually you are almost get the right the connection string, however you use the wrong prepended string. Further more detailed information you could refer to this doc:Configure connection strings.
Which string to use it depends on which type you choose, like in my test I use a custom type. Then I should use os.environ['CUSTOMCONNSTR_testconnectionstring'] to get the value.
From the doc you could find there are following types:
SQL Server: SQLCONNSTR_
MySQL: MYSQLCONNSTR_
SQL Database: SQLAZURECONNSTR_
Custom: CUSTOMCONNSTR_

I was struggling with this and found out this solution:
import os
setting = os.getenv("mysetting")

In Functions application settings, such as service connection strings, are exposed as environment variables during execution. You can access these settings by declaring import os and then using
setting = os.environ["mysetting"]
As Alex said, try to remove CONNECTIONSTRINGS: from the name of the environment variable. In azure portal just add mysetting in application settings as keyName.

I figured out the issue with help from George and selected George's answer as the correct answer.
I changed the code to os.environ["SQLCONNSTR_PDMPDBCONNECTIONSTRING"] but also I had been deploying the package from VS Code to Azure via the following code using Azure Command Line Interface (Azure CLI). This code overwrites Azure App Settings with the local.settings.json.
func azure functionapp publish <MY_FUNCTION_NAME> --publish-local-settings -i --overwrite-settings -y
I think that was causing changes to the type of database that I had specified in Azure App Settings (SQL Server), so when I had previously tried os.environ["SQLCONNSTR_PDMPDBCONNECTIONSTRING"] it didn't work because I was overwriting my Azure settings using my local.settings.json, which did not specify a database type.
I finally got it to work by deploying using the VS Code Extension called Azure Functions (Azure Functions: Deploy to Function App). This retains the settings that I had created in Azure App Services.

sqlanydb windows could not load dbcapi

I am trying to connect to a SQL Anywhere database via python. I have created the DSN and I can use command prompt to connect to the database using dbisql - c "DNS=myDSN". When I try to connect through python using con = sqlanydb.connect(DSN= "myDSN")
I get
`Traceback (most recent call last):
File "<pyshell#5>", line 1, in <module>
con = sqlanydb.connect(DSN= "RPS Integration")
File "C:\Python27\lib\site-packages\sqlanydb.py", line 522, in connect
return Connection(args, kwargs)
File "C:\Python27\lib\site-packages\sqlanydb.py", line 538, in __init__
parent = Connection.cls_parent = Root("PYTHON")
File "C:\Python27\lib\site-packages\sqlanydb.py", line 464, in __init__
'libdbcapi_r.dylib')
File "C:\Python27\lib\site-packages\sqlanydb.py", line 456, in load_library
raise InterfaceError("Could not load dbcapi. Tried: " + ','.join(map(str, names)))
InterfaceError: (u'Could not load dbcapi. Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)`

I was able to resolve the problem. I was never able to use the sqlanydb.connect. I ended up using pyodbc. So my final connection string was con = pydobc.connect(dsn="myDSN"). I think that sqlanydb is only fully functional with sqlanydb 17 and I was using a previous version.

Depending on how Sybase is locally installed, it could also be that the Python module really cannot find the (correct) library when looking up dbcapi.dll in your PATH. A quick fix for that is to create a new SQLANY_API_DLL environment variable (that's the initial None in the names the error message says it tried using; it takes priority over all the others) containing the correct path, usually something like %SQLANY16%\Bin64\dbcapi.dll depending on what version you have installed (Sybase usually creates an environment variable pointing to its installation folder on a per version basis).

One way to encounter the backtrace shown by the OP is running 64-bit Python interpreter, which is unable to load the 32-bit dbcapi.dll. Try launching with "py -X-32" to force a 32-bit engine, or reinstall using a 32-bit python engine.
Unfortunately sqlanydb's code that tries to be smart about finding dbcapi also ends up swallowing the exceptions thrown during loading. The sqlanydb author probably assumed that failure to load implies file not found, which isn't always the case.

I was stoked in this same problem, both windows system32 and 64.
I'm using Python 3.9 and I found that since Python 3.8 the cdll.LoadLibrary function does no longer search PATH to find library files.
To fix this, I created the enviroment variable SQLANY_API_DLL pointing to the route of sqlanywhere installation, in my case version 17, bin32 or bin64 depending on your system.

Connect to openstack is failing

I have written a bit of python code to interact with an Openstack instance; using the shade library.
The call
myinstance = shade.openstack_cloud(cloud='mycloud', **auth_data)
works fine on my local Ubuntu installation; but fails on our "backend" servers (running RHEL 7.2).
File "mystuff/core.py", line 248, in _create_connection
myinstance = shade.openstack_cloud(cloud='mycloud', **auth_data)
File "/usr/local/lib/python3.5/site-packages/shade-1.20.0-py3.5.egg/shade/init.py", line 106, in openstack_cloud
return OpenStackCloud(cloud_config=cloud_config, strict=strict)
File "/usr/local/lib/python3.5/site-packages/shade-1.20.0-py3.5.egg/shade/openstackcloud.py", line 312, in init
self._local_ipv6 = _utils.localhost_supports_ipv6()
File "/usr/local/lib/python3.5/site-packages/shade-1.20.0-py3.5.egg/shade/_utils.py", line 254, in localhost_supports_ipv6
return netifaces.AF_INET6 in netifaces.gateways()['default']
AttributeError: module 'netifaces' has no attribute 'AF_INET6'
The admin for that system tells me that IPv6 is not enabled there; maybe that explains the fail. I did some research, but couldn't find anything to prevent the failure.
Any thoughts are welcome.
Update: I edited my clouds.yml; and it looks like this:
# openstack/shade config file
# required to connect provisioning using the shade module
client:
force_ipv4: true
clouds:
mycloud:
auth:
user_domain_name: xxx
auth_url: 'someurl'
region_name: RegionOne
I also tried export OS_FORCE_IPV4=True - but the error message is still there.

If you go through the OpenStack os-client-config documentation, there they have mentioned about IPV6 related issue.
IPv6 is the future, and you should always use it if your cloud
supports it and if your local network supports it. Both of those are
easily detectable and all friendly software should do the right thing.
However, sometimes you might exist in a location where you have an
IPv6 stack, but something evil has caused it to not actually function.
In that case, there is a config option you can set to unbreak you
force_ipv4, or OS_FORCE_IPV4 boolean environment variable.
So, using these boolean config you can force appropriate network protocol. Adding below lines to your clouds.yaml file
client:
force_ipv4: true
will force IPV4 and hope it will resolve your issue.
Edit by OP: unfortunately the above doesn't help; fixed it by reworking shade-1.20.0-py3.5.egg/shade/_utils.py: I changed the return statement
return netifaces.AF_INET6 in netifaces.gateways()['default']`
to a simple
return False
And stuff is working. Of course, this is just a workaround; but a bug report was filed as well.

Using the paste "call" scheme when starting a Pyramid application

I have a Pyramid application which I can start using pserve some.ini. The ini file contains the usual paste configuration and everything works fine. In production, I use uwsgi, having a paste = config:/path/to/some.ini entry, which works fine too.
But instead of reading my configuration from a static ini file, I want to retrieve it from some external key value store. Reading the paste documentation and source code, I figured out, that there is a call scheme, which calls a python function to retrieve the "settings".
I implemented some get_conf method and try to start my application using pserve call:my.module:get_conf. If the module/method do not exist, I get an appropriate error, so the method seems to be used. But whatever I return from the method, I end up with this error message:
AssertionError: Protocol None unknown
I have no idea what return value of the method is expected and how to implement it. I tried to find documentation or examples, but without success. How do I have to implement this method?

While not the answer to your exact question, I think this is the answer to what you want to do. When pyramid starts, your ini file vars from the ini file just get parsed into the settings object that is set on your registry, and you access them through the registry from the rest of your app. So if you want to get settings somewhere else (say env vars, or some other third party source), all you need to do is build yourself a factory component for getting them, and use that in the server start up method that is typically in your base _ _ init _ _.py file. You don't need to get anything from the ini file if that's not convenenient, and if you don't, it doesn't matter how you deploy it. The rest of your app doesn't need to know where they came from. Here's an example of how I do this for getting settings from env vars because I have a distributed app with three separate processes and I don't want to be mucking about with three sets of ini files (instead I have one file of env vars that doesn't go in git and gets sourced before anything gets turned on):
# the method that runs on server startup, no matter
# how you deploy.
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application."""
# settings has your values from the ini file
# go ahead and stick things it from any source you'd like
settings['db_url'] = os.environ.get('DB_URL')
config = Configurator(
settings=settings,
# your other configurator args
)
# you can also just stick things directly on the registry
# for other components to use, as everyone has access to
# request.registry.
# here we look in an env var and fall back to the ini file
amqp_url = os.environ.get('AMQP_URL', settings['amqp.url'] )
config.registry.bus = MessageClient( amqp_url=amqp_url )
# rest of your server start up code.... below

Using Ladon in Python 2.6 on windows

I have been trying to create a web service out some python scripts, and haven't seemed to have had much luck. I am new to web services in general, but would really like to get this figured out. I'm on Windows 7 and use IIS7. The service also needs to be SOAP.
I've read through most posts that have anything to do with python and SOAP and tried out pretty much all the different libraries, but most of them just seem over my head (especially ZSI/SOAPpy). The Ladon Library seems like it would be best (and simplest) for what I need, but the tutorial http://www.youtube.com/watch?v=D_JYjEBedk4&feature=feedlik loses me at 5:10 when he brings it to the server. When I type the ladon2.6ctl in the cmd, it seems like windows gets quite confused. I'm guessing it is a little different because he is running on Linux and using Apache.
With that, any tips on how to get a python web service running on Microsoft 'stuff' would be greatly appreciated, as I have been trying to figure this stuff out for way too long.
One thing to note is the reason things are so specific (and seemingly strange) is because the scripts I have do a lot of geoprocessing with ESRI's "arcpy".
--Addition--
Traceback on localhost:8080/TestService:
Traceback (most recent call last):
<br>File "c:\Python26\ArcGIS10.0\lib\site-packages\ladon-0.5.1-py2.6.egg\ladon\server\wsgi_application.py", line 229, in __call__
exec("import %s" % ','.join(self.service_list))
File "<string>", line 1, in <module>
File "c:\Users\r\workspace\ladon\src\testspace.py", line 3, in <module>
class TestService2(object):
File "c:\Users\r\workspace\ladon\src\testspace.py", line 4, in TestService2
#ladonize(int,int,rtype=int)
File "c:\Python26\ArcGIS10.0\lib\site-packages\ladon-0.5.1-py2.6.egg\ladon\ladonizer\decorator.py", line 87, in decorator
ladon_method_info = global_service_collection().add_service_method(f,*def_args,**def_kw)
File "c:\Python26\ArcGIS10.0\lib\site-packages\ladon-0.5.1-py2.6.egg\ladon\ladonizer\collection.py", line 119, in add_service_method
sinfo = self.source_info(src_fname)
File "c:\Python26\ArcGIS10.0\lib\site-packages\ladon-0.5.1-py2.6.egg\ladon\ladonizer\collection.py", line 79, in source_info
a = ast.parse(src)
File "c:\Python26\ArcGIS10.0\lib\ast.py", line 37, in parse
return compile(expr, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 1
from ladon.ladonizer import ladonize
^
SyntaxError: invalid syntax
sample code:
from ladon.ladonizer import ladonize
class TestService2(object):
#ladonize(int,int,rtype=int)
def sum(self,a,b):
'''add two numbers<br>
param a: number 1
param b: number 2
rtype: sum of result
'''
return a+b

I must admit I normally use Linux for almost everything and I haven't tried Ladon on Windows for a while. I will spin up my windows installation later today and see if there is any trouble.
You wrote that ladon2.6ctl get's confused. Do you have an exception Traceback?

To summarize the fix for anyone else interested, delete the "syslog import" from these 3 ladon modules: ladon/interfaces/jsonwsp.py - line 6
ladon/dispatcher/dispatcher.py - line 7
ladon/server/wsgi_application.py - line 37
Then, you need to change the linefeed from window's default of /r/n to /n. In Eclipse, go to Window -> Preferences -> General, then select (not drop down) the Workspace tab. On the bottom right, select "other" under New text file line delimiter and change it to Unix.