I am trying to use Frama-c via python application. This python application sets some env variables and system path. From this application, I am calling Frama-c as a python process as following:
cmd = ['/usr/local/bin/frama-c', '-wp', '-wp-print', '-wp-out', '/home/user/temp','/home/user/project/test.c']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=False)
When this code is executed from Python application I am getting following error:
[kernel] Parsing FRAMAC_SHARE/libc/__fc_builtin_for_normalization.i (no preprocessing)
[kernel] warning: your preprocessor is not known to handle option ` -nostdinc'. If pre-processing fails because of it, please add -no-cpp-gnu-like option to Frama-C's command-line. If you do not want to see this warning again, use explicitely -cpp-gnu-like option.
[kernel] warning: your preprocessor is not known to handle option `-dD'. If pre-processing fails because of it, please add -no-cpp-gnu-like option to Frama-C's command-line. If you do not want to see this warning again, use explicitely -cpp-gnu-like option.
[kernel] Parsing 2675891095.c (with preprocessing)
[kernel] System error: /usr/bin/gcc -c -C -E -I. -dD -nostdinc -D__FC_MACHDEP_X86_32 -I/usr/local/share/frama-c/libc -o '/tmp/2675891095.cc8bf16.i' '/home/user/project/test.c': No child processes
I am finding it hard to debug what is causing the error:
System error: /usr/bin/gcc -c -C -E -I. -dD -nostdinc -D__FC_MACHDEP_X86_32 -I/usr/local/share/frama-c/libc -o '/tmp/2675891095.cc8bf16.i' '/home/user/project/test.c': No child
processes
Is there a way to generate more error log from Frama-c that might help me figure out the issue?
Note that this error only occur when I start the process(to execute Frama-c) from my application, and not if I start it from a python console. And it happens only on Linux machine and not on Windows machine.
Any help is appreciated. Thanks!!
Update :
I realized that by using -kernel-debug flag I can obtain stack trace. So I tried the option and get the following information:
Fatal error: exception Sys_error("gcc -E -C -I. -dD -D__FRAMAC__
-nostdinc -D__FC_MACHDEP_X86_32 -I/usr/local/share/frama-c/libc -o '/tmp/2884428408.c2da79b.i'
'/home/usr/project/test.c': No
child processes")
Raised by primitive operation at file
"src/kernel_services/ast_queries/file.ml", line 472, characters 9-32
Called from file "src/kernel_services/ast_queries/file.ml", line 517,
characters 14-26
Called from file "src/kernel_services/ast_queries/file.ml", line 703,
characters 46-59
Called from file "list.ml", line 84, characters 24-34
Called from file "src/kernel_services/ast_queries/file.ml", line 703,
characters 17-76
Called from file "src/kernel_services/ast_queries/file.ml", line 1587,
characters 24-47
Called from file "src/kernel_services/ast_queries/file.ml", line 1667,
characters 4-27
Called from file "src/kernel_services/ast_data/ast.ml", line 108,
characters 2-28
Called from file "src/kernel_services/ast_data/ast.ml", line 116,
characters 53-71
Called from file "src/kernel_internals/runtime/boot.ml", line 29,
characters 6-20
Called from file "src/kernel_services/cmdline_parameters/cmdline.ml",
line 787, characters 2-9
Called from file "src/kernel_services/cmdline_parameters/cmdline.ml",
line 817, characters 18-64
Called from file "src/kernel_services/cmdline_parameters/cmdline.ml",
line 228, characters 4-8
Re-raised at file "src/kernel_services/cmdline_parameters/cmdline.ml",
line 244, characters 12-15
Called from file "src/kernel_internals/runtime/boot.ml", line 72,
characters 2-127
And I looked at the file "src/kernel_services/ast_queries/file.ml", line 472 and the code executed is Sys.command cpp_command.
I am not sure why "No Child Processes" error is thrown when trying to execute execute gcc.
Update: I have Ocaml version: 4.02.3, Python version: 2.7.8 and Frama-C version: Silicon-20161101
I know nothing about Frama-C. However, the error message is coming from somebody's (OCaml's? Python's?) runtime, indicating that a system call failed with the ECHILD error. The two system calls that system() makes are fork() and waitpid(). It's the latter system call that can return ECHILD. What it means is that there's no child process to wait for. One good possibility is that the fork() failed. fork() fails when the system is full of processes (unlikely) or when a per-user process limit has been reached. You could check whether you're running up against a limit of this kind.
Another possibility that occurs to me is that some other part of the code is already handling child processes using signal handling (SIGCHLD). So the reason there's no child process to wait for is that it has already been handled elsewhere. This gets complicated pretty fast, so I would hope this isn't the problem.
Related
I'm doing a programming course (CS50W) and it has a command to submit the project to github automatically. It's called submit50.
When trying to push my code to github through this method, I do:
submit50 --verbose web50/projects/2020/x/capstone
And I'm getting the following error:
OSError: [Errno 7] Argument list too long: b'/usr/local/bin/git'
I have literally no idea what is happening here. If you could help me out I would be grateful.
Full error with traceback:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/bin/submit50", line 11, in <module>
load_entry_point('submit50==3.0.2', 'console_scripts', 'submit50')()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/submit50/__main__.py", line 147, in main
user_name, commit_hash, message = lib50.push("submit50", args.slug, CONFIG_LOADER, prompt=prompt)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lib50/_api.py", line 69, in push
with authenticate(remote["org"], repo=repo) as user, prepare(tool, slug, user, included):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lib50/_api.py", line 326, in prepare
_run(git(f"add -f {' '.join(shlex.quote(f) for f in included)}"))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lib50/_api.py", line 703, in _run
with _spawn(command, quiet, timeout) as child:
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lib50/_api.py", line 674, in _spawn
child = pexpect.spawn(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pexpect/pty_spawn.py", line 205, in __init__
self._spawn(command, args, preexec_fn, dimensions)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pexpect/pty_spawn.py", line 303, in _spawn
self.ptyproc = self._spawnpty(self.args, env=self.env,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pexpect/pty_spawn.py", line 315, in _spawnpty
return ptyprocess.PtyProcess.spawn(args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ptyprocess/ptyprocess.py", line 330, in spawn
raise exception
OSError: [Errno 7] Argument list too long: b'/usr/local/bin/git'
Submission cancelled.
"Argument list too long" is a Unix-style1 error that indicates that the arguments to the exec-like function are, well, too long. (The b'/usr/local/bin/git' part here is misleading: that's not the part that is too long.) The /Library/Frameworks/Python.framework/ prefix strongly suggests that you're on MacOS, which has a Mach-ish kernel that is Unix-based and has relatively small argument limits:
sh-3.2$ sysctl kern.argmax
kern.argmax: 262144
This means that there's a maximum of 256 kiB of argument characters, including the command itself and all the bytes of arguments, including the \0 byte that terminates each one. There's often a limit on the length of argv itself as well. The details vary from one system to another. See also this question on the unix.stackexchange.com site, for instance.
It's not immediately clear to me what to do about this. Looking at the stack trace, we find this pair of lines (which I've split up further for posting purposes here):
File "/Library/Frameworks/Python.framework/Versions/3.8/
lib/python3.8/site-packages/lib50/_api.py", line 326, in prepare
_run(git(f"add -f {' '.join(shlex.quote(f) for f in included)}"))
which points to the main culprit: a function named prepare in the lib50 library that is part of what your course provided. This is trying to run a Git command with many file names passed to its git add command. Where did it get the included list? That, we don't know. If this list were shorter, the problem would not occur, but if there is no way for you to shorten the list, knowing this won't help.
Raising kern.argmax would help, but it is probably not something you can set.
Using a different OS, with a different (or raise-able) maximum, could help.
The "right answer", though, is probably to have the software itself be smarter. There's no need to invoke git add this particular way, with every file listed as one big argv vector. Indeed, instead of invoking git add directly, a program should probably be invoking git update-index instead. A Python program should probably be using update-index with the --stdin and -z flags (and any other flags as appropriate for this particular function's intended usage).
1Other systems now provide Unix-like error numbers as well, even if they're not based on Unix. This particular one is E2BIG or error #7, on MacOS.
I need to invoke a powershell script and capture the output as generated from it.
Since I need to capture the output, I chose to use subprocess.run()
Powershell invocation
powershell DeleteResults -resultscsv '1111,2222,3333,4444'
Python(Python 3.5.2 :: Anaconda 4.1.1 (64-bit)) code
command = "powershell DeleteResults -resultscsv '{}'".format(resultscsv)
output = subprocess.run(command, stdout=subprocess.PIPE).stdout.decode('utf-8')
All goes fine as long as the length of command is less than 33K(approx)
However, subprocess.call() throws error when the length exceeds 33K
(There is no issue on the powershell side as it works perfectly fine when invoked directly)
ERROR: [WinError 206] The filename or extension is too long
Traceback (most recent call last):
File "D:\path\to\python\wrapper.py", line 154, in <module>
output = subprocess.run(command, stdout=subprocess.PIPE).stdout.decode('utf-8')
File "D:\Anaconda3\lib\subprocess.py", line 693, in run
with Popen(*popenargs, **kwargs) as process:
File "D:\Anaconda3\lib\subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "D:\Anaconda3\lib\subprocess.py", line 1224, in _execute_child
startupinfo)
Any pointer will be great help.
Not sure if relevant - the python script is invoked via Control-M on a windows environment.
--Edit--
Adding this section to add more details in response to answer by Alex.
We don't own the ps script DeleteResults. So, we can't modify it. We just consume it.
As it is done today,
The resultscsv(80K chars) is stored in a results.ini file
A small piece of ps inline code parses .ini file and then invokes DeleteResults. Note: There is powershell call inside the outer powershell invocation(invocation below).
This approach works perfectly fine even if chars >80K.
However, we don't want the inline ini parser to be part of invocation - looks ugly.
So, the idea is to write a python wrapper which will parse .ini file and invoke the powershell
powershell -NoLogo -NonInteractive -Command "Get-Content 'results.ini' | foreach-object -begin {$h=#{}} -process { $k = [regex]::split($_,'='); if(($k[0].compareTo(' ') -ne 0) -and ($k[0].startswith('[') -ne $True)) {$h.Add($k[0], $k[1]) }}; powershell DeleteResults -resultscsv $h.Get_Item('resultscsv')"
So, wondering why the above ps 1-liner not hitting the char length limit ? Is it that the line powershell DeleteResults -resultscsv $h.Get_Item('resultscsv') is NOT actually expanded inline - thereby not hitting the char length limit ?
There is command line string limitation, it's value depends on OS version.
It is not possible to pass large data through command line arguments. Pass a filename instead.
Documentation and workaround https://support.microsoft.com/en-us/help/830473/command-prompt-cmd-exe-command-line-string-limitation
I'm experimenting porting a project from Cherrypy to Pyramid web framework. I have a small part converted and notice that Ctrl+C will not stop the Pyramid application. A cookiecutter version will stop with Ctrl+C. I end up needing to kill the process every time.
I am serving using the pserve command that uses the waitress WSGI server in both cases...
pserve development.ini
I should also note: I am running Debian Stretch in a VirtualBox VM.
Is there a way to know why the behavior has changed or how to restore Ctrl+C shutdown? How could I know if something is now blocking this from happening?
-- Additional Information asked for in comments --
Using grep Sig /proc/process_id/status yields the following:
SigQ: 0/15735
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000180004002 hex/binary 110000000000000000100000000000010
Using GDB and getting a py-bt
(gdb) py-bt
Traceback (most recent call first):
<built-in method select of module object at remote 0x7f914f837e58>
File "/usr/lib/python3.5/asyncore.py", line 144, in poll
r, w, e = select.select(r, w, e, timeout)
File "/usr/lib/python3.5/asyncore.py", line 203, in loop
poll_fun(timeout, map)
File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/waitress/server.py", line 131, in run
use_poll=self.adj.asyncore_use_poll,
File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/waitress/__init__.py", line 17, in serve
server.run()
File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/waitress/__init__.py", line 20, in serve_paste
serve(app, **kw)
File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/paste/deploy/util.py", line 55, in fix_call
val = callable(*args, **kw)
File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/paste/deploy/loadwsgi.py", line 189, in server_wrapper
**context.local_conf)
File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/pyramid/scripts/pserve.py", line 239, in run
server(app)
File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/pyramid/scripts/pserve.py", line 32, in main
return command.run()
File "/home/clutton/programs/python/webapps_pyramid/env/bin/pserve", line 11, in <module>
sys.exit(main())
In order to diagnose where I was running into issues I took the following steps guided by many of the comments made on the question.
I inspected the process to ensure that signals were indeed being caught.
grep Sig /proc/process_id/status
Which yields the following information:
SigQ: 0/15735
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000180004002 hex/binary 110000000000000000100000000000010
The SigCgt indicates signals that are indeed being listened to, in the above the hex value converted to binary shows that (from right to left) signal 2 and 15 were indeed bound.
At this point we need to diagnose why there would be a handler yet it appeared to not be working. So the remaining question was what was the handler. To find that out I used the Python signal module and added some code where I could see it in a debugger...
import signal
s = signal.getsignal(2)
Once I did that I found that the handler referenced a function from a standalone script that is part of the project. I was overwriting the default signal handlers in order to do cleanup before terminating the process but... I was importing it as well in part of this project that had it's own process. Since the project is normally developed on Windows, I was probably dealing with different signals when using Ctrl-C previously so this bug has existed for a long time and doing some Linux development work for the project brought it to light.
I have a Python program that is directly executed by Jenkins. This program uses the sh library to execute a docker container, via this function. Note that it is an important feature of this function that it display the subprocess's output as it executes:
def run_command(*args, **kwargs):
# pass the parent stream 'tty' state to the command:
tty_in = sys.stdin.isatty()
tty_out = sys.stdout.isatty()
run = sh.Command(args[0])
try:
for line in run(args[1:], _err=sys.stdout, _iter=True, _tty_in=tty_in, _tty_out=tty_out):
sys.stdout.write(line)
sys.stdout.flush()
As per the comments, docker run requires a TTY for input, so the keyword argument _tty_in is set to match whatever the state of stdin is. However when running under Jenkins, it is False.
However the issue is around UTF-8 encoded error messages from programs running within the container such as cp. This results in errors such as:
cp: cannot stat \xe2\x80\x98filename...
It turns out those three bytes are utf-8 encoding for a special "quote" character that cp likes to use when the locale is UTF-8. If I set the locale to "C" manually before running cp directly, I can see that it uses normal ascii instead.
When my Python script encounters these errors, it dies with the following:
Exception in thread Thread-10:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1484, in output_thread
done = stream.read()
File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1974, in read
self.write_chunk(chunk)
File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1949, in write_chunk
self.should_quit = self.process_chunk(chunk)
File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1847, in process
handler.write(chunk)
File "/usr/lib/python2.7/codecs.py", line 351, in write
data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 16: ordinal not in range(128)
This suggests to me that the sh module is expecting ascii output from the subprocess, but is receiving UTF-8 and is unable to decode it.
I found the _encoding and _decode_errors options for sh and although these do seem to affect the locale that cp sees when run directly by sh, it does not appear to translate correctly for programs running within the docker container. However it does allow the program to continue as the decoding errors are skipped rather than raising an exception.
I would prefer to understand the situation better and implement a proper solution. Can anyone explain what is actually going on here, step by step (Jenkins > Python > sh > Docker > Bash)
I'm using Python 2.7.12 with Jenkins 2.33.
I had a similar problem when running python script in docker container.
I solved the following procedure.
(Step 1) add the following line in Dockerfile before running python command
ENV LANG C.UTF-8
I am trying to implement the dependency parser using stanford-corenlp-python wrapper.
Since I am using Windows Platform which gives error for pexpect package while execution, I spend some time to install wexpect for Windows and follow the steps here and completed the setup. Now when I am trying to execute corenlp.py again, I am getting following error and program is getting terminated.
Please help me with this.
Traceback (most recent call last):
File "corenlp.py", line 258, in <module>
nlp = StanfordCoreNLP()
File "corenlp.py", line 169, in __init__
self.corenlp.expect("done.", timeout=20) # Load pos tagger model (~5sec)
File "/cygdrive/f/Masters-Spring2015/Natural Language Processing/Project/stanford-corenlp-python/wexpect.py", line 1356, in expect
return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
File "/cygdrive/f/Masters-Spring2015/Natural Language Processing/Project/stanford-corenlp-python/wexpect.py", line 1370, in expect_list
return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize)
File "/cygdrive/f/Masters-Spring2015/Natural Language Processing/Project/stanford-corenlp-python/wexpect.py", line 1441, in expect_loop
raise EOF (str(e) + '\n' + str(self))
wexpect.EOF: End Of File (EOF) in read_nonblocking(). Empty string style platform.
<wexpect.spawn_unix object at 0x7fdad40c>
version: 2.3 ($Revision: 399 $)
command: /cygdrive/c/Windows/system32/java
args: ['/cygdrive/c/Windows/system32/java', '-Xmx1800m', '-cp', './stanford-corenlp-full-2014-08-27/stanford-corenlp-3.4.1.jar:./stanford-corenlp-full-2014-08-27/stanford-corenlp-3.4.1-models.jar:./stanford-corenlp-full-2014-08-27/joda-time.jar:./stanford-corenlp-full-2014-08-27/xom.jar:./stanford-corenlp-full-2014-08-27/jollyday.jar', 'edu.stanford.nlp.pipeline.StanfordCoreNLP', '-props', 'default.properties']
searcher: searcher_re:
0: re.compile("done.")
buffer (last 100 chars):
before (last 100 chars): not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
after: <class 'wexpect.EOF'>
match: None
match_index: None
exitstatus: None
flag_eof: True
pid: 7104
child_fd: 3
closed: False
timeout: 30
delimiter: <class 'wexpect.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
The stanford-corenlp-python wrapper starts a server which spawns an instance of the command line interface for CoreNLP, receives sentences on HTTP, pipes sentences to the spawned CLI instance via stdin, takes the stdout result, parses it, and sends back sentences via HTTP, then parses it again in the process which sent the request in the first place.
The error you're seeing--that wexpect is getting an unexpected EOF--looks like the spawned instance of the CoreNLP CLI is crashing. This is probably because you're running this in Cygwin. Cygwin looks like unix, but when it needs to do really unix-y things, like run other programs, things that require genuine interaction with the operating system, it starts to turn into garbage.
I'm guessing you went Cygwin because you can't use pexpect on the Windows command line, and the stanford-corenlp-python wrapper uses it. From pexpect's docs, though:
Pexpect only works on POSIX systems, where the pty module is present in the standard library. It may be possible to run it on Windows using Cygwin.
It think we're seeing an instance here where pexpect fails on Cygwin.
My recommendation: don't use the stanford-corenlp-python wrapper; it is slow and buggy. Even after you get this working, something else will be wrong, and even after you get it running, it will drag out the processing tremendously.
Instead, run CoreNLP directly from the command line. If you have a batch job to do, use a filelist. The output is XML; you can parse that / grep it, and be on your way. From the CoreNLP home page:
If you want to process a list of files use the following command line:
java -cp stanford-corenlp-VV.jar:stanford-corenlp-VV-models.jar:xom.jar:joda-time.jar:jollyday.jar:ejml-VV.jar -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP [ -props <YOUR CONFIGURATION FILE> ] -filelist <A FILE CONTAINING YOUR LIST OF FILES>
Let me tell you about my own experience using the wrapper. The last time I used the stanford-corenlp-python wrapper, it took 17800 documents two weeks to process. That was after a good week or two of tweaking the code to handle the large quantity of news articles I was about to pass it.
Later, I got a new computer and ran CoreNLP through the command line directly, no wrapper except some python and shell scripting to help create a target file list for the batch jobs. It took 24000 documents 15 hours process. The new computer helped, but I anticipated maybe a 4x performance gain from the new hardware. The other 26x was from dropping the wrapper.
Considering how the stanford-corenlp-python wrapper is written, it's not surprising that it's slow and buggy. Between all the crazy layers of processing it does, I'm amazed they got it working at all. You're better off, though, just running on the command line, especially since you won't have to run it in Cygwin.