Develop Reference

How do I stop git from asking credentials when I try to clone a repository that doesn't exist?

I'm doing some research and I need to download a very large number of git repositories, something like 17k+.
I wrote a very simple Python script to automate the cloning routine from a dataset containing the GitHub URLs.
first_10 = data.url
name = data.project_name
for x, i in zip(first_10, name):
os.system('git clone {} D:\gitres\{}'.format(x,i))
It just iterates over those two pandas dataframe columns for the URL to be cloned and the folder name.
Here comes the problem: every time the script finds a URL that no longer exists on GitHub, the script halts its routine, asks for credentials and won't resume until my input. Doesn't matter if I input correct credentials or gibberish, it will do this every time it finds an invalid GitHub URL. How do I stop git from asking those credentials?

The reason GitHub sends you a 401 to prompt for credentials if the repository is missing is because they don't want to leak whether a private repository exists. If they didn't prompt, you could easily determine that the repository does exist by getting a 401 and that it doesn't by getting a 404. Instead, GitHub always prompts for credentials, and only then returns a 404 if the repository doesn't exist or isn't accessible to you.
If your desire is not to be prompted at all, as torek mentioned, you can simply set the environment variable GIT_ASKPASS to false and this will work. You could also set GIT_TERMINAL_PROMPT to 0 and that would prevent any prompting for credentials.
However, I strongly recommend that you do indeed set some credentials because GitHub will much more aggressively rate-limit you if you don't set any credentials, and if you do end up using an excessive amount of resources, it's much easier for GitHub to contact you about the problem and ask you to fix it, rather than just block you or make an abuse report to your network provider.
On that note, your Python script is not likely to handle the case where you have a large number of failures for that reason, so you should strongly consider handling that case most robustly. In general, anyone making a large number of HTTP requests to any server needs to learn to gracefully back off.
If you decide you do want to pass credentials, you can do so from the environment using a custom credential helper, or you can use an SSH key and SSH URLs to do this.

I'd suggest you've overspecified the problem (turning this into an XY problem): you don't specifically need to make Git not ask for credentials since you could instead merely clone those repositories that do exist.
That said, there are two ways to prevent Git from asking for credentials:
Use a URL that cannot take credentials. (Any given server may or may not accept such URLs. With GitHub, you could try to log in as git#github.com via ssh, and present a valid public key. After ssh has authenticated you, Git will give you access to any accessible URL, and deny you access to any inaccessible or invalid URL, without asking for further credentials.)
Supply a credential helper that never actually provides any credentials, without asking for any. For instance, you could run with GIT_ASKPASS=false in the environment. See the credentials documentation for details.
(There's one more option as well, which is to allow Git to ask for credentials but redirect the input to a program. This is trickier than just overriding GIT_ASKPASS so there is no reason to cover it here.)
To solve the problem better, find a way to list out the repositories that do exist, and do not attempt to clone the ones that don't. This is likely to go significantly faster.

My guess is that you are using https:// data urls. If you use a personal access token, then GitHub shouldn't be asking you for a username/password. Take a look here on how to set it up.
I think that if you use ssh:// data urls instead, then you wouldn't encounter that problem because git defaults to using your ssh-key for authentication instead of password.

You probably want to check that the repo exists before attempting the clone. There are answers on Stack Overflow for this here.
Alternatively, if you switch to using subprocess instead of os.system, you can simply "trick" it by reading input from /dev/null which will prevent the prompt - that way, no intervention will be needed and the invalid URLs will simply be skipped.
for x, i in zip(first_10, name):
subprocess.call(['git', 'clone', x, i], shell=False, stdin=subprocess.DEVNULL, start_new_session=True)

I have come across another useful trick that might come in handy working in desktop environments.
in conjunction with using GIT_TERMINAL_PROMPT=0, git -c credential.helper= <rest of commands> helps to also suppress credential manager windows to pop up.

Change the owner of a file from a user to root using sudoers in linux

I want to use /etc/sudoers to change the owner of a file from bangtest(user) to root.
Reason to change: when I uploaded an image from bangtest(user) to my server using Django application then image file permission are like
ls -l /home/bangtest/alpha/media/products/image_2093.jpg
-rw-r--r-- 1 bangtest bangtest 28984 May 6 02:47
but when I tried to access those file from server using //myhost/media/products/image_2093.jpg, I am getting 404 error.When I tried to log the error its like
Caught race condition abuser. attacker: 0, victim: 502 open file owner: 502, open file: /home/bangtest/alpha/media/products/image_2093.jpg
After when I changed the owner of a file from bangtest to root,then I am able to access the image perfectly.
So because of that reason I want to change owner of file dynamically using python script.
I have tried by changing the sudoers file like mentioned below.But still I am getting error like
chown: changing ownership of `image.jpg': Operation not permitted
My sudoers code:
root ALL=(ALL) ALL
bangtest ALL=(ALL) /bin/chown root:bangtest /home/bangtest/alpha/*
Any Clues why sudoers are not working?
Note:Operating system Linux.
Thanks

It looks to me like your problem is that you're trying to access a file as a file on your local machine by using //myhost (a path to it). // won't work with a server, if that's what you're going for, or at least it won't with the ruby on rails ones i'm using. if it's a location on your local machine use //myhost/media/products/image_2093.jpg and if it's something you're trying to access through your server, try /myhost/media/products/image_2093.jpg. I don't know of any good posts to refer you to but I just tested it out for myself.
Hope it helps
I was able to find some interesting info that sounds like evidence.
http://www.mail-archive.com/dev#httpd.apache.org/msg55666.html

Use ACL
setfacl -dm u::rwx,g::rwx,o::rwx /home/bangtest/alpha/media/products/
All files created in this folder have permissions set to rwx for all users.
EDIT:// Change permissions for chmod is dangerous, but is possible, edit /etc/sudoers
bangtest ALL=(ALL) NOPASSWD: /usr/bin/chmod

howto get PAM authentication working with Apache and mod_authnz_external?

I'm trying for 2 days now to write a custom PAM script which authenticates under Linux with sys user and pass.
I made examples in C, Python and Perl but I have the same problem with all of them.
When I test the script in my shell everything works well.
I test them with
# ./script;echo $?
And get 0 or 1 back.
But as soon as I try to use it with mod_authz_external and Apache it stops working.
Even with a valid user I get an 1 back and are not able to log in.
It seems that there is a problem with PAM and maybe the Apache environment.
I read somewhere that I have to set the environment variables to use PAM but I have no clue how.
Here are examples in C, Perl and Python:
C: http://pastebin.com/v9Yn9xvK
Perl: http://pastebin.com/cqzqztYg
Python: http://pastebin.com/32cvvCjS
Choose whatever you like, they work all the same... returning the exit code 0 or 1.
Every help or hint would be appreciated.
Thanks!

If you're authenticating a system user with PAM, then it's going to go through libpam_unix.so. From the manpage of pam_unix:
A helper binary, unix_chkpwd(8), is provided to check the user's password when it is stored in a read protected database. This binary is very simple and will only check the password of the user invoking it.
So if you want to authenticate users from a webserver, you're most likely running as user 'apache' or something like that. All pam_unix can do for you is authenticate user 'apache', which is most likely not what you want. pwauth must somehow bypass this restriction.

How should I store login FTP credentials inside a python file which is executable by other unix users?

The question is, imagine that I want to create a deploy script which uses 'fabric' deploy library, which has to specify the FTP credentials where you want to deploy to. The idea is that I would like to store this script in our testing server, and from that server, it will deploy remotely to another servers. I would like to create a user account to each developer, but I don't want to share with them the FTP credentials, but rather, give them only the executable, so, if I create a python executable and I added to /user/bin for instance, they will be able to execute it, but also making a 'which mycommand' they can see the source where is inside the credentials, what can I do to avoid it?
Thanks!!

If you care about security, you probably should be using scp or sftp instead. These can be set up to not require any keystrokes, while still having decent security. For more see: http://www.debian-administration.org/articles/152
However, if you really want/need to use ftp, you probably should put the credentials in a file and chmod it to mode 400: r-------- or perhaps 440: r--r-----. Embedding credentials in a script isn't a great thing.

Put the credentials in a file to which the individual developers have no access.
Create an account that DOES have access to that file but DOES NOT allow interactive logons. - Create your FTP submission program, make it runnable by this second account.
Put all the developers in a group (e.g. "Devs").
Add an entry in the sudoers file to allow members of the "Devs" group to run the FTP program without additional authentication. This will be something like:
%Devs ALL=(ALL) ALL, , NOPASSWD: /path/to/FTPscript

How to programmatically(python) authenticate username/password using OS users

I'm writing a python program which allow user to login to it. I don't want to implement my own authentication but would rather take advantage of the OS(linux) mechanism. That is, when the user is trying to sign in my app by inputing username/password pair(which should be a valid OS user), I need to authenticate the pair by the OS. How to do that ? It may need the subprocess module, yet I've tried with no luck.

Try using PAM via Python PAM or similar

That should be possible by having your script read the /etc/passwd and /etc/shadow files, which contain details about usernames and passwords on a Linux system. Do note that the script will have to have read access to the files, which depending on the situation may or may not be possible.
Here are two good articles explaining the format of those files, which should tell you everything you need to know in order to have your script read and understand them:
Understanding /etc/passwd File Format
Understanding /etc/shadow File Format
By the way, when it talks about encrypted password, it means that it has been encrypted using the DES algorithm. You'll probably need to use pyDes or another python implementation of the DES algorithm in order for your script to create an encrypted password that it can compare to the one in /etc/shadow.