Develop Reference

How to ensure python binaries are on your path?

I am trying to use the Kaggle api. I have downloaded kaggle using pip and moved kaggle.json to ~/.kaggle, but I haven't been able to run kaggle on Command Prompt. It was not recognized. I suspect it is because I have not accomplished the step "ensure python binaries are on your path", but honestly I am not sure what it means. Here is the error message when I try to download a dataset:
>>> sys.version
'3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)]'
>>> import kaggle
>>> kaggle datasets list -s demographics
File "<stdin>", line 1
kaggle datasets list -s demographics
^
SyntaxError: invalid syntax

kaggle is python module but it should also install script with the same name kaggle which you can run in console/terminal/powershell/cmd.exe as
kaggle datasets list -s demographics
but this is NOT code which you can run in Python Shell or in Python script.
If you find this script kaggle and open it in editor then you can see it imports main from kaggle.cli and it runs main()
And this can be used in own script as
import sys
from kaggle.cli import main
sys.argv += ['datasets', 'list', '-s', 'demographics']
main()
But this method sends results directly on screen/console and it would need assign own class to sys.stdout to catch this text in variable.
Something like this:
import sys
import kaggle.cli
class Catcher():
def __init__(self):
self.text = ''
def write(self, text):
self.text += text
def close(self):
pass
catcher = Catcher()
old_stdout = sys.stdout # keep old stdout
sys.stdout = catcher # assing new class
sys.argv += ['datasets', 'list', '-s', 'demographics']
result = kaggle.cli.main()
sys.stdout = old_stdout # assign back old stdout (because it is needed to run correctly `print()`
print(catcher.text)
Digging in source code on script kaggle I see you can do the same using
import kaggle.api
kaggle.api.dataset_list_cli(search='demographics')
but this also send all directly on screen/console.
EDIT:
You can get result as list of special objects which you can later use with for-loop
import kaggle.api
result = kaggle.api.dataset_list(search='demographics')
for item in result:
print('title:', item.title)
print('size:', item.size)
print('last updated:', item.lastUpdated)
print('download count:', item.downloadCount)
print('vote count:', item.voteCount)
print('usability rating:', item.usabilityRating)
print('---')

Is it possible to let the entire R Studio program run from Python? [duplicate]

I searched for this question and found some answers on this, but none of them seem to work. This is the script that I'm using in python to run my R script.
import subprocess
retcode = subprocess.call("/usr/bin/Rscript --vanilla -e 'source(\"/pathto/MyrScript.r\")'", shell=True)
and I get this error:
Error in read.table(file = file, header = header, sep = sep, quote = quote, :
no lines available in input
Calls: source ... withVisible -> eval -> eval -> read.csv -> read.table
Execution halted
and here is the content of my R script (pretty simple!)
data = read.csv('features.csv')
data1 = read.csv("BagofWords.csv")
merged = merge(data,data1)
write.table(merged, "merged.csv",quote=FALSE,sep=",",row.names=FALSE)
for (i in 1:length(merged$fileName))
{
fileConn<-file(paste("output/",toString(merged$fileName[i]),".txt",sep=""))
writeLines((toString(merged$BagofWord[i])),fileConn)
close(fileConn)
}
The r script is working fine, when I use source('MyrScript.r') in r commandline. Moreover, when I try to use the exact command which I pass to the subprocess.call function (i.e., /usr/bin/Rscript --vanilla -e 'source("/pathto/MyrScript.r")') in my commandline it works find, I don't really get what's the problem.

I would not trust too much the source within the Rscript call as you may not completely understand where are you running your different nested R sessions. The process may fail because of simple things such as your working directory not being the one you think.
Rscript lets you directly run an script (see man Rscript if you are using Linux).
Then you can do directly:
subprocess.call ("/usr/bin/Rscript --vanilla /pathto/MyrScript.r", shell=True)
or better parsing the Rscript command and its parameters as a list
subprocess.call (["/usr/bin/Rscript", "--vanilla", "/pathto/MyrScript.r"])
Also, to make things easier you could create an R executable file. For this you just need to add this in the first line of the script:
#! /usr/bin/Rscript
and give it execution rights. See here for detalis.
Then you can just do your python call as if it was any other shell command or script:
subprocess.call ("/pathto/MyrScript.r")

I think RPy2 is worth looking into, here is a cool presentation on R-bloggers.com to get you started:
http://www.r-bloggers.com/accessing-r-from-python-using-rpy2/
Essentially, it allows you to have access to R libraries with R objects that provides both a high level and low level interface.
Here are the docs on the most recent version: https://rpy2.github.io/doc/latest/html/
I like to point Python users to Anaconda, and if you use the package manager, conda, to install rpy2, it will also ensure you install R.
$ conda install rpy2
And here's a vignet based on the documents' introduction:
>>> from rpy2 import robjects
>>> pi = robjects.r['pi']
>>> pi
R object with classes: ('numeric',) mapped to:
<FloatVector - Python:0x7fde1c00a088 / R:0x562b8fbbe118>
[3.141593]
>>> from rpy2.robjects.packages import importr
>>> base = importr('base')
>>> utils = importr('utils')
>>> import rpy2.robjects.packages as rpackages
>>> utils = rpackages.importr('utils')
>>> packnames = ('ggplot2', 'hexbin')
>>> from rpy2.robjects.vectors import StrVector
>>> names_to_install = [x for x in packnames if not rpackages.isinstalled(x)]
>>> if len(names_to_install) > 0:
... utils.install_packages(StrVector(names_to_install))
And running an R snippet:
>>> robjects.r('''
... # create a function `f`
... f <- function(r, verbose=FALSE) {
... if (verbose) {
... cat("I am calling f().\n")
... }
... 2 * pi * r
... }
... # call the function `f` with argument value 3
... f(3)
... ''')
R object with classes: ('numeric',) mapped to:
<FloatVector - Python:0x7fde1be0d8c8 / R:0x562b91196b18>
[18.849556]
And a small self-contained graphics demo:
from rpy2.robjects.packages import importr
graphics = importr('graphics')
grdevices = importr('grDevices')
base = importr('base')
stats = importr('stats')
import array
x = array.array('i', range(10))
y = stats.rnorm(10)
grdevices.X11()
graphics.par(mfrow = array.array('i', [2,2]))
graphics.plot(x, y, ylab = "foo/bar", col = "red")
kwargs = {'ylab':"foo/bar", 'type':"b", 'col':"blue", 'log':"x"}
graphics.plot(x, y, **kwargs)
m = base.matrix(stats.rnorm(100), ncol=5)
pca = stats.princomp(m)
graphics.plot(pca, main="Eigen values")
stats.biplot(pca, main="biplot")

The following code should work out:
import rpy2.robjects as robjects
robjects.r.source("/pathto/MyrScript.r", encoding="utf-8")

I would not suggest using a system call to there are many differences between python and R especially when passing around data.
There are many standard libraries to call R from Python to choose from see this answer

If you just want run a script then you can use system("shell command") of the sys lib available by import sys. If you have an usefull output you can print the result by " > outputfilename" at the end of your shell command.
For example:
import sys
system("ls -al > output.txt")

Try adding a line to the beginning of your R script that says:
setwd("path-to-working-directory")
Except, replace the path with the path to the folder containing the files features.csv and BagofWords.csv.
I think the problem you are having is because when you run this script from R your working directory is already the correct path, but when you run the script from python, it defaults to a working directory somewhere else (likely the top of the user directory).
By adding the extra line at the beginning of your R script, you are explicitly setting the working directory and the code to read in these files will work. Alternatively, you could replace the filenames in read.csv() with the full filepaths of these files.
#dmontaner suggested this possibility in his answer:
The process may fail because of simple things such as your working directory not being the one you think.

c++ scoring program using python for the class

I am trying to make python program to score the homework written in c++.
My laptop os is window so i installed g++ in order to use subprocess command.
This is my code.
Python code:
import subprocess
import os
import glob
from openpyxl import Workbook
path='C:\\auto_scoring\\'
def write_csv(tc_num,ans):
score_book=Workbook()
def score(testcases):#scroing function
correct=True
for i,testcase in enumerate(testcases):
if correct:
print(i+1,":correct")
else :
print(i+1,":wrong")
def exe_maker(filename):
ret=filename.rstrip('.cpp')+'.exe'
return ret
def execute(file):
ret=subprocess.check_output('check')
def get_testcase():
global path
text_name='test.txt'
r=open(text_name,mode='r',encoding='utf8')
text=r.read()
print(text)
def run():
file_list = glob.glob('*.cpp*')#cpp
for file in file_list:
exe_file=exe_maker(file)
command='g++ -o '+path+exe_file+" "+path+file
print(command)
subprocess.call(command)
print("start!")
ret=subprocess.check_output(file.rstrip('.cpp')+" 1").decode('ascii')
print(ret)
def main():
run()
def exp():
ret=subprocess.check_output('check').decode('ascii')
if ret=='1':
print('correct')
else:
print('wrong')
if __name__=='__main__':
main()
But I have to use pass the parameter such as
C++ example:
int main(int argc,char*argv){
cout<<argv[1];
}
I don't want to use this kind of thing. Program without int argc and char argv.
I just want to check whether the answer is correct.
For example) input: My name is k -> output: Hello K.
If i couldn't make the program , I have to run every 100 c++ files...
I really need help.
thank you.

Try using subprocess.run, where you pass each item in your command line as a list:
command=['g++', '-o' , exe_file, path + file]
subprocess.call(command)

How do I get the Python line number and file name of the point this function was called from? [duplicate]

In C++, I can print debug output like this:
printf(
"FILE: %s, FUNC: %s, LINE: %d, LOG: %s\n",
__FILE__,
__FUNCTION__,
__LINE__,
logmessage
);
How can I do something similar in Python?

There is a module named inspect which provides these information.
Example usage:
import inspect
def PrintFrame():
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename) # __FILE__ -> Test.py
print(info.function) # __FUNCTION__ -> Main
print(info.lineno) # __LINE__ -> 13
def Main():
PrintFrame() # for this line
Main()
However, please remember that there is an easier way to obtain the name of the currently executing file:
print(__file__)

For example
import inspect
frame = inspect.currentframe()
# __FILE__
fileName = frame.f_code.co_filename
# __LINE__
fileNo = frame.f_lineno
There's more here http://docs.python.org/library/inspect.html

Building on geowar's answer:
class __LINE__(object):
import sys
def __repr__(self):
try:
raise Exception
except:
return str(sys.exc_info()[2].tb_frame.f_back.f_lineno)
__LINE__ = __LINE__()
If you normally want to use __LINE__ in e.g. print (or any other time an implicit str() or repr() is taken), the above will allow you to omit the ()s.
(Obvious extension to add a __call__ left as an exercise to the reader.)

You can refer my answer:
https://stackoverflow.com/a/45973480/1591700
import sys
print sys._getframe().f_lineno
You can also make lambda function

I was also interested in a __LINE__ command in python.
My starting point was https://stackoverflow.com/a/6811020 and I extended it with a metaclass object. With this modification it has the same behavior like in C++.
import inspect
class Meta(type):
def __repr__(self):
# Inspiration: https://stackoverflow.com/a/6811020
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
# print(info.filename) # __FILE__ -> Test.py
# print(info.function) # __FUNCTION__ -> Main
# print(info.lineno) # __LINE__ -> 13
return str(info.lineno)
class __LINE__(metaclass=Meta):
pass
print(__LINE__) # print for example 18

wow, 7 year old question :)
Anyway, taking Tugrul's answer, and writing it as a debug type method, it can look something like:
def debug(message):
import sys
import inspect
callerframerecord = inspect.stack()[1]
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename, 'func=%s' % info.function, 'line=%s:' % info.lineno, message)
def somefunc():
debug('inside some func')
debug('this')
debug('is a')
debug('test message')
somefunc()
Output:
/tmp/test2.py func=<module> line=12: this
/tmp/test2.py func=<module> line=13: is a
/tmp/test2.py func=<module> line=14: test message
/tmp/test2.py func=somefunc line=10: inside some func

import inspect
.
.
.
def __LINE__():
try:
raise Exception
except:
return sys.exc_info()[2].tb_frame.f_back.f_lineno
def __FILE__():
return inspect.currentframe().f_code.co_filename
.
.
.
print "file: '%s', line: %d" % (__FILE__(), __LINE__())

Here is a tool to answer this old yet new question!
I recommend using icecream!
Do you ever use print() or log() to debug your code? Of course, you
do. IceCream, or ic for short, makes print debugging a little sweeter.
ic() is like print(), but better:
It prints both expressions/variable names and their values.
It's 40% faster to type.
Data structures are pretty printed.
Output is syntax highlighted.
It optionally includes program context: filename, line number, and parent function.
For example, I created a module icecream_test.py, and put the following code inside it.
from icecream import ic
ic.configureOutput(includeContext=True)
def foo(i):
return i + 333
ic(foo(123))
Prints
ic| icecream_test.py:6 in <module>- foo(123): 456

To get the line number in Python without importing the whole sys module...
First import the _getframe submodule:
from sys import _getframe
Then call the _getframe function and use its' f_lineno property whenever you want to know the line number:
print(_getframe().f_lineno) # prints the line number
From the interpreter:
>>> from sys import _getframe
... _getframe().f_lineno # 2
Word of caution from the official Python Docs:
CPython implementation detail: This function should be used for internal and specialized purposes only. It is not guaranteed to exist in all implementations of Python.
In other words: Only use this code for personal testing / debugging reasons.
See the Official Python Documentation on sys._getframe for more information on the sys module, and the _getframe() function / submodule.
Based on Mohammad Shahid's answer (above).

Example of use libextractor3 from python

I'm using python-extractor to work with libextractor3. I can not find any examples of it. Does any one have any documentations or examples?

Source package of python-extractor contains a file named extract.py which has a small demo on how to use libextractor Python binding.
Content from extract.py
import extractor
import sys
from ctypes import *
import struct
xtract = extractor.Extractor()
def print_k(xt, plugin, type, format, mime, data, datalen):
mstr = cast (data, c_char_p)
# FIXME: this ignores 'datalen', not that great...
# (in general, depending on the mime type and format, only
# the first 'datalen' bytes in 'data' should be used).
if (format == extractor.EXTRACTOR_METAFORMAT_UTF8):
print "%s - %s" % (xtract.keywordTypes()[type], mstr.value)
return 0
for arg in sys.argv[1:]:
print "Keywords from %s:" % arg
xtract.extract(print_k, None, arg)
To have better understanding of python-extractor go through source code in extractor.py.