Is there a similar config to that of .condarc (anaconda 4.0.0) that allows Jupyter to be configured to work behind a corporate proxy on a local machine?
Error received:
HTTPError: HTTP Error 407: Proxy Authentication Required
Way easier: Just add the following to your notebook:
In [1]: import os
os.environ['http_proxy'] = "http://user:passwd#host:port"
os.environ['https_proxy'] = "https://user:passwd#host:port"
after that, requests will work OK=200, e.g.
In [2]: import requests
requests.get("https://google.com")
Out[2]: <Response [200]>
Based on this link.
You have to modify the Jupyter notebook server env. Create a file named 00-something.py under your Jupyter notebook server profile and add the following:
For example:
vi /.jupyter/profile_myserver/startup/00-startup.py
(or on Windows open up "C:/Users/your username/.jupyter/profile_myserver/startup/00-startup.py" in your editor of choice)
and add
import sys,os,os.path
os.environ['HTTP_PROXY']="http://proxy.example.com:80"
os.environ['HTTPS_PROXY']="https://proxy.example.com:443"
you can confirm the env variables by running
%env
in a cell and the output
{'CLICOLOR': '1',
'GIT_PAGER': 'cat',
'HOME': '/home/jay',
'HTTP_PROXY': 'http://proxy.example.com:80',
..
Next try
import requests
requests.get("http://google.com")
If you get a response [200] then you are all set.
Use the lowercase variable instead, it works for me:
import sys,os,os.path
os.environ['http_proxy']="http://user:passwd#host:port"
os.environ['https_proxy']="http://user:passwd#host:port"
Then check your env variable using this:
%env
The output will be like this:
{'CLICOLOR': '1',
'...'
'...'
'http_proxy': 'http://gunawan.marbun:xxxxxxxx#cache.itb.ac.id:8080'
'https_proxy': 'https://gunawan.marbun:xxxxxxxx#cache.itb.ac.id:8080'
'no_proxy': 'localhost,127.0.0.0/8,::1'}
Notes: Since I can't comment due to my reputation (req 50 and I'm newbie), I present new answer instead.
An easier solution for me was to add an exception to my proxy configuration. I just put the address http://localhost:8888 to my exception list and it worked.
Based on these Jupyter customization instructions:
Create the directory .jupyter_config in your home directory
Add the line JUPYTER_CONFIG_DIR=~/.jupyter_config to your bash/shell profile (eg. .bash_profile).
Add a script titled startup.py to ~/.jupyter_config with the following code, customized with your specific proxy information:
import os
os.environ['http_proxy']= "http://user:passwd#host:port"
os.environ['https_proxy']= "https://user:passwd#host:port"
os.environ['HTTP_PROXY']= os.environ['http_proxy']
os.environ['HTTPS_PROXY']= os.environ['https_proxy']
1. Ensure you are connected to VPN to office network
2. You can set up proxy before starting notebook or in every notebook
A. before starting
from jupyer notebok, start terminal (From option New on right top)
set HTTP_PROXY=http://fakeserver:fakeport
set HTTPS_PROXY=http://fakeserver:fakeport
jupyter notebook
(This will start notebook in new terminal, with proxy set that can be checked with %env)
B. in every notebook
import sys,os,os.path
os.environ['HTTP_PROXY']="http://fakeserver:fakeport"
os.environ['HTTPS_PROXY']="http://fakeserver:fakeport"
3. in windows, set the environmental variables
This PC (Rt Click) >> propertis>> Advanced System Setting >> Advanced >> Envioronmental variables>> click NEW to add 2 variables pointing HTTP_PROXY and HTTPS_PROXY to "http://fakeserver:fakeport"
Mostly the problem comes due to adding HTTPS in the proxy server URL. For both HTTP and HTTPS, the proxy server URL should be "http://" only
import os
os.environ['http_proxy'] = "http://username:complexpassword!#34#serverIP:port"
os.environ['https_proxy'] = "http://username:complexpassword!#34#serverIP:port"
import requests
requests.get("https://google.com")
Response should be 200 OK
If you are in a Windows domain and use domain\username for authentication, just use username instead of domain\username in python setting above
For those of you who may need a socks5 proxy, the following did the trick for me:
os.environ['http_proxy'] = "socks5h://host:port"
os.environ['https_proxy'] = "socks5h://host:port"
The only difference with the http proxy was the socks5h part in the url. In my case the proxy didn't have username and password but I suppose these too may be added to the url like socks5h://user:passwd#host:port.
Related
I'm running through Sendgrid's intro material for Python but executing the example code throws a 403-Forbidden error.
Steps I took:
Create API Key & sendgrid.env file as instructed.
Create a conda environment with python 3.5: conda create -n sendgrid python=3.5
Install sendgrid: (sendgrid) pip install sendgrid
Run example: (sendgrid) python main.py
Where main.py contains the exact code copied from the example page linked above.
Issue: Running main.py throws the error HTTP Error 403: Forbidden.
Things I've tried:
I tried switching out the to and from emails in that example, but doesn't change the result.
I also tried the same flow but using NodeJS but same result.
Any ideas on what I'm doing wrong?
Give API Key its full access, follow steps:
Settings
API Keys
Edit API Key
Full Access
Update
Whitelist your domain, follow steps:
Settings
Sender Authentication
Domain Authentication
Select DNS Host
Enter your domain name
Copy all records and put them in your Advanced DNS management console
NOTE: When adding records, make sure not to have domain name in the host. Crop it out.
If you do not want to authenticate domain, you can try with Single Sender Verification as well.
Note: It might take some time for records to start functioning.
If you're using pylinter, e.message will say
Instance of 'Exception' has no 'message' member
This is because message attribute is generated dynamically by sendgrid which pylinter is unable to access as it doesn't exists before runtime.
So, to prevent that, at the top of your file or above print(e.message) line, you need to add either one of the below, they mean the same thing-
# pylint: disable=no-member
E1101 is code to no-member, fine more here
# pylint: disable=E1101
Now the code below should work for you. Just make sure you have SENDGRID_API_KEY set in environment. If not, you may also directly replace it with os.environ.get("SENDGRID_API_KEY") which is not a good practice though.
# pylint: disable=E1101
import os
from sendgrid import SendGridAPIClient
from sendgrid.helpers.mail import Mail
message = Mail(
from_email="from_email#your-whitelisted-domain.com",
to_emails=("recipient1#example.com", "recipient2#example.com"),
subject="Sending with Twilio SendGrid is Fun",
html_content="<strong>and easy to do anywhere, even with Python</strong>")
try:
sg = SendGridAPIClient(os.environ.get("SENDGRID_API_KEY"))
response = sg.send(message)
print(response.status_code)
print(response.body)
print(response.headers)
except Exception as e:
print(e.message)
I am trying to connect a basic mail sender sendgrid function to my program's pipeline. Problem is when I do necessery commands from terminal it works fine like :
echo "export SENDGRID_API_KEY='YOUR_API_KEY'" > sendgrid.env
echo "sendgrid.env" >> .gitignore
source ./sendgrid.env
python3 sendgrid_mail.py
but when I try to run it from PyCharm it gives me HTTP Error 401: Unauthorized
error.
import os
from sendgrid import SendGridAPIClient
from sendgrid.helpers.mail import Mail
def mail_sender():
message = Mail(
from_email='from_mail',
to_emails='to_mail',
subject='hello pycharm',
html_content='<strong>and easy to do anywhere, even with Python</strong>')
try:
sg = SendGridAPIClient(os.environ.get('my_api_key'))
response = sg.send(message)
print(response.status_code)
print(response.body)
print(response.headers)
except Exception as e:
print(e)
mail_sender()
It's unclear how you're running in PyCharm. I'm guessing your environment variable is not actually set while you are in PyCharm. Try looking at the API key and see if it's what you're expecting (Either just print or use the debugger)
You can modify the environment variables in your run configuration
Run | Edit Configurations... | Select the run configuration you're using | Environment variables ....
You could also try setting the environment variable prior to starting PyCharm and it might pick it up, but that depends on a-lot of other things.
You're getting the environment variable my_api_key, but you're setting your environment variable as SENDGRID_API_KEY in the sendgrid.env. These two names need to match.
I'm assuming YOUR_API_KEY in your sendgrid.env file is the api key you've created. If not, that would need to be replaced with a real value.
I have some Python 3 code and can make it use the module slackclient to post to channels, no problem. However if I run this code from our corporate servers where all traffic needs to go through a proxy it fails. I know the proxy server and port and have to use them to run pip from our servers, like this:
pip install --proxy proxy.evilcorp.com:8080 slackclient
That works great. If I don't proxy the pip, it fails to connect as expected. So that tells me I just need to figure out how to get my slackclient code to use the proxy, but how? Here is my code:
from slackclient import SlackClient
def get_slackclient():
token = "blah-blah-token"
sc = SlackClient(token)
return sc
def post_slackmessage(username,channel,text):
sc = get_slackclient()
try:
sc.api_call("chat.postMessage",channel=channel,text=text,username=username,unfurl_links="true")
except:
print ("failed to post messaage to slack")
post_slackmessage("test_slack", "test", "hurrah it posted")
I just can't seem to figure out where to put proxy settings, I must be missing something simple. I'm open to other outside the box ideas to make this all work but I can't really install anything on the server to make all traffic go through the proxy or change the proxy settings.
Figured it out. I'll leave this here in case somebody else has the same problem. Looks like it's built in, just pass a proxies dict in.
def get_slackclient():
#https://api.slack.com/custom-integrations/legacy-tokens
token = "blah-blah-blah"
proxies = dict(https="proxy.evilcorp.com:8080", http="proxy.evilcorp.com:8080")
sc = SlackClient(token, proxies=proxies)
return sc
Well, that was easy :)
UPDATE
If you happen to upgrade to the latest slack module, it is a little different and only http:// proxies are supported (no secure for you!). And you pass in a str instead of a dict so just one proxy.
Just change to this:
proxy = "proxy.evilcorp.com:8080"
sc = slack.WebClient(token, timeout=60, proxy=proxy)
You'll note that actually making the call to the api has changed as well, like this:
sc.chat_postMessage(channel=thechannel, text=thetext, username=theusername, unfurl_links="true")
I am trying to create a very simple Python script to download the contents of an internal service at my company that sits within our firewall and authenticates using kerberos.
When I installed the requests_kerberos module I first edited the import kerberos in it to use import kerberos_sspi as kerberos instead after having installed the kerberos_sspi module.
Thus I have the following Python script
import requests
from requests_kerberos import HTTPKerberosAuth
response = requests.get('http://service.internaldomain',auth=HTTPKerberosAuth())
print response
While trying to process the 401 it crashes out with the error.
error: (-2146893053, 'InitializeSecurityContext', 'The specified target is unknown or unreachable')
While looking into seeing if I could do this with curl instead I ran kinit and noticed that it asked me for the password to authorisation with the following prompt:
Password for username#additionalInternalDomain.internaldomain
Thus I wondered if this might be what is causing the issue.
I have tried multiple libraries on python and failed when trying to authenticate from a windows machine.There is no easy way. The Kerberos libraries mainly work on Linux. The workarounds for Windows do not work. So what can be the solution to this.
Well... be a Roman while in Rome. Try the windows native libraries from Python.
import sys
import clr
from System.Net.Http import *
myClienthandler = HttpClientHandler()
myClienthandler.UseDefaultCredentials = True
myClient = HttpClient(myClienthandler)
x = myClient.GetStringAsync("putyourURLwithinthequoteshere")
myresult = x.Result
print(myresult)
Note that the this python script will have to run by the user who has access to the URL you are trying to access. By setting UseDefaultCredentials property as True, you are passing the Kerberos tickets for the logged in user.
The server is giving you a 401 challenge - and the client (usually a browser or even curl) provides the credentials in a subsequent call. If you are already logged in at your domain - try forcing a pre-emptive hop, i.e. you’d carry your Kerberos ticket with your call and the server will not give you a 401 challenge:
kerberos_auth = HTTPKerberosAuth(force_preemptive=True)
r = requests.get("http://myhost/DXAPIGraphQL/api/graphql", auth=kerberos_auth)
If the above doesn't help look into the:
principal and hostname_override arguments of the HTTPKerberosAuth class.
I had to connecto to a REST API who's in a keberized environment just now.
After some reading, i came to this (and it worked):
tk = 'long_kerberos_token'
headers = {'Authorization': 'Negotiate' + tk}
r = requests.get(url=PING_URL, headers=headers)
Since I don't have enough reputation I could answer the following question:
I'm trying to connect to TestLink via the xmlrpc API. I've set the following in TestLink's config.inc.php:
$tlCfg->api->enabled = TRUE;
$tlCfg->exec_cfg->enabled_test_automation = ENABLED;
and restarted the apache sever. I tried to connect the TestLink server via the python package TestLink-API-Python-client (https://github.com/orenault/TestLink-API-Python-client)
from testlink import TestlinkAPIClient, TestLinkHelper
import sys
URL = 'http://MYSERVER/testlink/lib/api/xmlrpc.php'
DevKey = 'MYKEY'
tl_helper = TestLinkHelper()
myTestLink = tl_helper.connect(TestlinkAPIClient)
myTestLink.__init__(URL, DEVKEY)
myTestLink.checkDevKey()
And then I receive a TLConnectionError, stating my url, and 404 Not Found...
Does anyone have any idea?
Thanks.
ANSWER: Remove "testlink" from the URL. For example:
URL = 'http://MYSERVER/lib/api/xmlrpc.php'
Edit testlinkhelper.py file and change the line
"DEFAULT_SERVER_URL = http://localhost/testlink/lib/api/xmlrpc/v1/xmlrpc.php"
and replace the localhost with your server ip and check the URL in the browser.
I was able to get the following to work with TestLink 1.9.3 (My company uses an older version - I think the current TL version is 1.9.12). I'm using the latest Python TestLink API version 0.5.2. Please change your server url and dev key for the following code. Also note that I use https instead of http to connect to the xmlrpc file.
import testlink
TESTLINK_API_PYTHON_SERVER_URL="https://testlink/etc/etc/etc"
TESTLINK_API_PYTHON_DEVKEY="someapikey"
tls = testlink.TestLinkHelper( TESTLINK_API_PYTHON_SERVER_URL, TESTLINK_API_PYTHON_DEVKEY ).connect(testlink.TestlinkAPIClient)
print tls.countProjects()
tc_info = tls.getTestCase(None, testcaseexternalid='S-10000')
print tc_info
print tls.whatArgs('createTestPlan')
I also posted it as an issue here