I am attempting to connect to a solr server using this tutorial. At this point, I am confident that my solr is set up correctly. I am able to run
> solr start -p 8983
and it appears to start something up.
sure enough
> solr status
Solr process 31421 running on port 8983
So now in my python code, I try what I think should be a basic connection script.
import solr
host = "http://localhost:8983/solr"
# also tried
# host = "http://localhost:8983"
# also tried
# host = "http://127.0.0.1:8983/solr"
# also tried
# host = "http://127.0.0.1:8983"
connection = solr.SolrConnection(host)
try:
connection.add(
id= 1,
title= "Lucene in Action",
author= ['Zack', 'Hank Hill']
)
except Exception as e:
import pdb
pdb.set_trace()
connection.commit()
My code never makes it to the connection.commit(), instead, it hits the debug point in the exception. Looking at exception e
HTTP code=404, Reason=Not Found, body=<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/update. Reason:
<pre> Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
So it looks like the python client is not finding the solr server, due to the 404? This seems like it should be pretty simple, so I'm not sure where I messed up here. Can anyone point me in the right direction?
edit : I added this script to check various hosts, no go
hosts = [
'http://localhost:8983/solr',
'http://localhost:8983',
'http://127.0.0.1:8983/solr',
'http://127.0.0.1:8983'
]
def connect(host):
connection = solr.SolrConnection(host)
try:
connection.add(
id= 1,
title='Lucene in Action',
author= ['Zack Botkin', 'Hank Hill']
)
except:
raise
for host in hosts:
try:
connect(host)
except Exception as e:
import pdb
pdb.set_trace()
Each exception is the same, 404 error
edit 2 : I was able to
> telnet localhost 8983
and it connected, so I'm pretty sure the solr server is running on that port?
To index with solr you will need to also create a core and make sure to use that core in your url. For example, once solr has been started run this command to create a core named test:
solr create -c test
Once that has been created you should see it listed in the solr admin page. To use it you can simply add that core name to your connection url. Simple example python code:
import solr
# create a connection to a solr server
s = solr.SolrConnection('http://localhost:8983/solr/test')
# add 2 documents to the index
s.add(id=1, title='Lucene in Action', author=['bob', 'asdf'])
s.add(id=2, title='test2', author=['Joe', 'test'])
s.commit()
# do a search
response = s.query('joe')
for hit in response.results:
print hit['title']
More information here https://cwiki.apache.org/confluence/display/solr/Running+Solr
Related
I have to get the list(+ some parameters) of all the VDIs(Virtual Desktop Interfaces) which are connected to a Citrix Broker.
In PowerShell, I do it like this:
...
# credentials used for remote connection to citrix controllers
$pass=$(ConvertTo-SecureString -String $env:CitrixPassword -AsPlainText -Force)
[pscredential]$creds = New-Object System.Management.Automation.PSCredential ($env:CitrixUser,$pass)
#create the session
$session = New-PSSession -ComputerName $controller -Credential $creds -Authentication Negotiate
Write-Output -message "Session on $controller successfully established!"
Write-Output -message "loading citrix snapin"
Invoke-Command -Session $session -ScriptBlock {Add-PSSnapin citrix*} # pay attention here
Write-Output -message "Loading Snapin successful"
#get the stuff
Write-Output -message "Read the data..."
$controllers = Invoke-Command -Session $session -ScriptBlock {Get-BrokerController}
Write-Output -message "... controllers done"
$desktops = Invoke-Command -Session $session -ScriptBlock {Get-BrokerDesktop -MaxRecordCount 10000}
Write-Output -message "... desktop done"
...
I am struggling since some hours to find a solution for python, and I was testing a few stuff with python but nothing seems to be working. I have mostly played with WSman but I start to feel that this is not the right way... still not sure though.
My test looks like that:
...
# create the session
wsman = WSMan(citrix_ddc,
username = citrix_user,
password = citrix_pass,
auth = "basic",
port = 443,
cert_validation = False)
with RunspacePool(wsman) as pool:
ps = PowerShell(pool)
ps.add_cmdlet("Add-PSSnapin").add_parameter("citrix*")
ps.invoke()
# we will print the first object returned back to us
print(ps.output[0])
The endpoint seems to be wrong: https://<my_controller_ip>:443/wsman and the error is:
Code: 404, Content: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Found</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Found</h2>
<hr><p>HTTP Error 404. The requested resource is not found.</p>
</BODY></HTML>
This should be pretty similar with connecting with CmdLets to VMware or Nutanix, but I never did it with python and it seems that I can't figure out how to do it.
Can someone please help me to find a python based solution to this?
Additional info:
the script will run in a docker container on a linux based machine.
I have finally found an answer. The wsman is the right way to go, but when I was debugging the working powershell script using a debugger, I found out what I did wrong.
I should not be using SSL at all # this might not be the same for other situations, but it is for my particular case
because of previous mention, I had to remove the WSMan port
In PS script I was using auth=negotiate, therefore I did the same in Python.
All the rest the same, and it worked.
Here is my working code:
# create the session
wsman = WSMan(citrix_controller_ip,
username = citrix_user,
password = citrix_pass,
ssl = False,
auth = "negotiate",
encryption = 'never'
)
with RunspacePool(wsman) as pool:
ps = PowerShell(pool)
ps.add_cmdlet('Add-PSSnapin').add_parameter('Name', 'Citrix*')
ps.add_statement()
ps.add_cmdlet("Invoke-Expression").add_parameter("Command", "Get-BrokerDesktop -MaxRecordCount 1000000 | ConvertTo-Json -Compress")
ps.add_cmdlet("Out-String").add_parameter("Stream")
ps.invoke()
print(json.loads(ps.output[0])
I have a Django app below that uses a proxy to connect to an external Postgres database. I had to replace another package with psycopg2 and it works fine locally, but doesn't work when I move onto our production server which is a Heroku app using QuotaguardStatic for proxy purposes. I'm not sure what's wrong here
For some reason, the psycopg2.connect part returns an error with a different IP address. Is it not inheriting the proxy set in the context manager? What would be
from apps.proxy.socks import Socks5Proxy
import requests
PROXY_URL = os.environ['QUOTAGUARDSTATIC_URL']
with Socks5Proxy(url=PROXY_URL) as p:
public_ip = requests.get("http://wtfismyip.com/text").text
print(public_ip) # prints the expected IP address
print('end')
try:
connection = psycopg2.connect(user=EXTERNAL_DB_USERNAME,
password=EXTERNAL_DB_PASSWORD,
host=EXTERNAL_DB_HOSTNAME,
port=EXTERNAL_DB_PORT,
database=EXTERNAL_DB_DATABASE,
cursor_factory=RealDictCursor # To access query results like a dictionary
) # , ssl_context=True
except psycopg2.DatabaseError as e:
logger.error('Unable to connect to Illuminate database')
raise e
Error is:
psycopg2.OperationalError: FATAL: no pg_hba.conf entry for host "12.345.678.910", user "username", database "databasename", SSL on
Basically, the IP address 12.345.678.910 does not match what was printed at the beginning of the context manager where the proxy is set. Do I need to set a proxy another method so that the psycopg2 connection uses it?
I am having trouble getting a Python CGI script to execute properly on an apache2 webserver that is running on a virtual Ubuntu 18 server. The hosting provider is DreamCompute, if that matters. I have gotten CGI scripts working under var/www/html/apps/cgi-bin/. Both helloworld.py and helloworld.pl execute fine both in the SSH terminal and in browser (Microsoft Edge).
The script giving me trouble is a Python script that accesses a MySQL database, reads data from it, then uses that data to fill some lists and generate some simple output at random (a magical spell with random effects for a tabletop RPG).
The spell generator script also executes fine in SSH, but when I try to view it in a browser it throws a 500 internal server error. The apache error logs tell me that the problem is end of script output before headers. I looked around, but couldn't find anyone with a similar problem and configuration.
EDIT: The full entry in the error log is:
[Tue Apr 20 00:20:25.324101 2021] [cgid:error] [pid 17275:tid 140105176721152] [client 2607:fea8:1d41:b800:7dca:305:b11a:447f:62505] End of script output before headers: spell_generator.py, referer: http://my.website/apps/cgi-bin/
After adding and removing parts of the Python script to see how it behaves in a browser, I believe I have isolated the problem: the part of the script that connects to the MySQL database. That database is also hosted on the same Ubuntu virtual machine, and it definitely has the right kind of data in it (just strings and IDs; nothing fancy).
Here's the Python code. I've removed some documentation comments but it should be pretty straightforward:
#!/usr/bin/python3
print("Content-type: text/html\n\n");
# The above two lines are boilerplate to ensure no print statements cause
# problems later in the program.
# Import statements
import cgi
import cgitb
cgitb.enable();
from random import choice
import mysql.connector
import sys
from enum import Enum
# Initialize enums for system comparison
class Ruleset(Enum):
GENERIC = "GENERIC"
GENERIC_DB = "generic_attributes"
DND_5 = "DND5e"
DND_5_DB = "dnd5e_attributes"
PATHFINDER_1 = "PF1e"
PATHFINDER_1_DB = "pathfinder1e_attributes"
# Determine what system to use, generic by default
spellSystem = Ruleset.GENERIC.value;
if (len(sys.argv)) > 1:
if (sys.argv[1] == Ruleset.DND_5.value):
spellSystem = Ruleset.DND_5.value;
if (sys.argv[1] == Ruleset.PATHFINDER_1.value):
spellSystem = Ruleset.PATHFINDER_1.value;
# === PROBLEM HERE ===
# Initialize SQL cursor, defaulting to generic
if spellSystem == Ruleset.DND_5.value:
spellDB = mysql.connector.connect(
host = "localhost",
user = "RemoteUser",
password = "password",
database = Ruleset.DND_5_DB.value
)
elif spellSystem == Ruleset.PATHFINDER_1.value:
spellDB = mysql.connector.connect(
host = "localhost",
user = "RemoteUser",
password = "password",
database = Ruleset.PATHFINDER_1_DB.value
)
else:
spellDB = mysql.connector.connect(
host = "localhost",
user = "RemoteUser",
password = "password",
database = Ruleset.GENERIC_DB.value
)
spellCursor = spellDB.cursor();
spellCursor.execute("SELECT ElementName FROM Element");
listHolder = spellCursor.fetchall();
# === CODE BELOW DOES NOT CAUSE PROBLEMS ===
#
# [logic that uses the SQL data]
#
# Output HTML page
print("<html> <head> <title>TEST - Magic Spell Generator</title> <link rel='stylesheet' href='../../css/style.css'> </head>");
print("<body> body contents");
print("</body>");
print("</html>");
The RemoteUser is a user account on the SQL server, which is used by "external" (non-root) programs to access the databases. password in the actual deployed script is a cryptographically secure password.
I'm not a Python expert, but the code runs with no problems when I execute it from the SSH terminal, so I don't think that bad code is to blame (although I could certainly be wrong). Here's a list of things I've tried already:
Checking for too many active SQL connections (there appears to be only one, the root user).
Making sure the script file has the correct privileges (chmod 755, same as the rest).
Making sure the necessary Python modules are installed (they are, and the Python-MySQL connector is the up to date one that works with the version of Python I'm using).
Restarting apache2.
I've spent most of today trying to find an answer. Any help or potential leads are welcome.
Turns out the problem wasn't actually the script trying to access the SQL database, which I figured out when I ran it properly in the SSH terminal (./spell_generator.py instead of python3 spell_generator.py). It crashed due to segmentation fault (core dumped), which implied that it was a code problem rather than a configuration problem.
After a bit of searching, I found Database connectivity through MySQL connector Python with CGI is not working, which pointed me to the real culprit: I was importing modules in the wrong order. I changed the import statements so that import mysql.connector is now the first one, and the script runs perfectly.
It shouldn't make a difference, but it does. I'm just happy.
I have written Jython script to create a mail session in IBM websphere.
Jython Script :
import sys
nodeName =sys.argv[0]
serverName =sys.argv[1]
def createSession(nodeName,serverName):
print "Creating mailsession"
ds =AdminConfig.getid('/Node:'+nodeName+'/Server:'+serverName+'/MailProvider:Built-in Mail Provider/')
print ds
print AdminConfig.required('MailSession')
name = ['name','MailSession']
jndi = ['jndiName','mail/Session']
host = ['mailTransportHost','mailhost.misys.global.ad']
storehost = ['mailStoreHost','mailhost.misys.global.ad']
mailAttrs=[name,jndi,host,storehost]
print mailAttrs
ss = AdminConfig.create('MailSession',ds,mailAttrs)
AdminConfig.save()
After running the script i am able to see mail session created by script in console. but it is throwing an error on server as below :
[Root exception is javax.naming.NameNotFoundException: Context: MyServer20Cell/nodes/MyServer20Node/servers/MyServer20,
name: mail/Session: First component in name mail/Session not found.
But the strange thing is when i opened the IBM Console and go to mail Session , without modifying any value in mail session, click on apply changes ,save it and restart the server .It Works fine and server is not throwing any error.
Can any one tell did i have done anything wrong in Script. How i can resolve this issue.
This problem is solved. need to add this property - mailTransportProtocol and add it in attributes
I'm getting a 'need to login' error when trying to interact with my MongoHQ database through python console on heroku:
...
File "/app/.heroku/venv/lib/python2.7/site-packages/pymongo/helpers.py", line 128, in _check_command_response
raise OperationFailure(msg % response["errmsg"])
pymongo.errors.OperationFailure: command SON([('listDatabases', 1)]) failed: need to login
My applicable code
app/init.py:
from mongoengine import connect
import settings
db = connect(settings.DB, host=settings.DB_HOST, port=settings.DB_PORT, username=settings.DB_USER, password=settings.DB_PASS)
app/settings.py:
if 'MONGOHQ_URL' in os.environ:
url = urlparse(os.environ['MONGOHQ_URL'])
DB = url.path[1:]
DB_HOST = url.hostname
DB_PORT = url.port
DB_USER = url.username
DB_PASS = url.password
os.environ['MONGOHQ_URL'] looks like:
'mongodb://[username]:[password]#[host]:[port]/[db-name]'
This code works (connects and can read and write to mongodb) both locally and from the heroku web server.
According to the docs (http://www.mongodb.org/display/DOCS/Connections), it should at make a 'login' attempt on connection to the server as long as the username and password params are passed to Connection or parseable from the URI. I couldn't think of a way to see if the login attempt was being made and failing silently.
I've tried bypassing mongoengine and using pymongo.Connection and got the same result. I tried all of the several patterns of using the Connection method. I created a new database user, different from the one mongoHQ creates for heroku's production access -> same same.
It's a flask app, but I don't think any app code is being touched.
Update
I found a solution, but it will cause some headaches. I can manually connect to the database by
conn = connect(settings.DB, host=settings.DB_HOST, port=settings.DB_PORT, username=settings.DB_USER, password=settings.DB_PASS)
db = conn[settings.DB]
db.authenticate(settings.DB_USER, settings.DB_PASS)
Update #2
Mongolab just worked out of the box.
Please use the URI method for connecting and pass the information to via the host kwarg eg:
connect("testdb_uri", host='mongodb://username:password#localhost/mongoenginetest')
MongoHQ add-on uses password hashes not actual passwords and that's perhaps the error.
You should change the environment variable MONGOHQ_URL to a real password with the following command:
heroku config:set MONGOHQ_URL=mongodb://...
Once set, you may restart your applications (heroku apps) so the change gets picked up. If you're in the directory of the failing application, config:seting the config var will restart the application.