Comparison of Python modes for Emacs - python

So I have Emacs 24.3 and with it comes a quite recent python.el file providing a Python mode for editing.
But I keep reading that there is a python-mode.el on Launchpad, and comparing the two files it jumps out to me that the former is under 4000 lines, while the latter is almost 20000. This suggests that the latter is much more feature-rich.
And I can't find any online feature comparison about them, documentation, or at least a list about the features for each of them. Yep, there is syntax highlighting and embedded interpreter, but what about completion in shell buffer, completion in source file buffer, autoindent, reindent etc.
So what are the important features of these modes? (Or any other Python mode for Emacs which you recommend.) Please provide detailed answers.

I was python-mode.el user once but quit using it a year ago because I felt the way it was developed was not well organized. Here is a list from the note I took at that time. But I need to warn you that almost a year is passed since then so the situation may be changed.
Many copy & paste functions.
Many accidentally working code. For example, not passing around variables but using implicit binding. This produces many compile errors (and will not work if you change it to lexical scope).
Rough granularity of commit. I send a patch, and it was committed with unrelated changes.
One thing I like about python-mode.el is it comes with automated test set (although I've never run it). python.el does not have a test set yet. But I know the author of python.el is writing it now.
While python.el is compact, it does not mean you get poor functionality. It is more like keeping core small and let others to extend it by providing concise API. Same author of python.el wrote python-django.el to extend python.el for django projects. I wrote auto-completion plugin for Python called Jedi.el and advanced IPython plugin called EIN. Both of them have better support for python.el than python-mode.el (well, that's because I don't use python-mode.el, though).
I had a few thing I missed from python-mode.el first, but they are quickly fixed in python.el (Of course, this probably means that I was not using so much functionality in python-mode.el).
what about completion in shell buffer, completion in source file buffer, autoindent, reindent etc.
completion in shell buffer:
It sort of works in both python.el and python-mode.el. But sometimes it does not work if you have bad combination of Emacs version and python(-mode).el version. So probably python.el is safer in this manner.
But if you want better solution, use EIN :)
completion in source file buffer:
Just use Jedi.el :)
autoindent/reindent:
I don't know which one is better in performance-wise. However, keybind for return differs one to the other. In python-mode.el, if you type RET you get autoindent. In python.el, RET does not give you indentation and you should use C-j instead. Actually C-j for newline+indentation is universal behavior in Emacs. So python.el is better if you do programming in other languages.

Being heavily involved in developing python-mode.el last years, my comment probably is biased: Recommend to stay with python.el for Emacs beginners. Also its author deserves credit for some useful approaches.
python-mode.el is designed to boost edits productivity. It makes it easy to run or execute via python2 and python3 or IPython shells in parallel.
It reduces number of needed keystrokes providing tailored commands. It makes edits faster, assists programming by speech, macro-driven input etc.
Supports Python language features not known from python.el currently:
py-up, py-down - travelling nested blocks
Avoid typos fetching forms at point, a clause for example:
py-backward-clause
py-copy-clause
py-down-clause
...
No need to customize when testing different versions:
py-execute-clause-python2
py-execute-clause-python3
py-execute-clause-ipython
...
notion of finer grained parts - py-expression, py-minor-expression
commands running versioned and paralleled (I)Python-executables, no need to re-define the default Python
largely removing the need of an active region marked before, see py-execute-line and a whole bunch more
To get an overview, have a look at the menu. Directory "doc" lists commands.
As the quality of code is raised: a way to compare both modes probably is checking for the bugs listed in http://debbugs.gnu.org/. See for example bug #15510, #16875; or http://lists.gnu.org/archive/html/help-gnu-emacs/2014-04/msg00250.html
Commented at "rough granularity of commit" already: While tkf basically is right looking for smaller pieces, sometimes conditions make me leave the rule. Considerable parts are not written by hand, but by programs residing in directory "devel". They create files used in development branch frist - i.e. components-python-mode. When starting a new feature it's often not obvious if the path chosen is fruitful.
After a hundred would-be-commits or so it still might turn out impossible or not so recommendable. Instead of posting all the meanders, used to keep that experimental branch for several days in these cases and check in when tests passed.
BTW assume tkf refers not to compile-errors --which would be looked for instantly-- but compiler warnings. Unfortunately Emacs mixes warning about backed style preferences with real errors.

Related

Encrypt python script [duplicate]

I am developing a piece of software in Python that will be distributed to my employer's customers. My employer wants to limit the usage of the software with a time-restricted license file.
If we distribute the .py files or even .pyc files it will be easy to (decompile and) remove the code that checks the license file.
Another aspect is that my employer does not want the code to be read by our customers, fearing that the code may be stolen or at least the "novel ideas".
Is there a good way to handle this problem?
"Is there a good way to handle this problem?" No. Nothing can be protected against reverse engineering. Even the firmware on DVD machines has been reverse engineered and the AACS Encryption key exposed. And that's in spite of the DMCA making that a criminal offense.
Since no technical method can stop your customers from reading your code, you have to apply ordinary commercial methods.
Licenses. Contracts. Terms and Conditions. This still works even when people can read the code. Note that some of your Python-based components may require that you pay fees before you sell software using those components. Also, some open-source licenses prohibit you from concealing the source or origins of that component.
Offer significant value. If your stuff is so good -- at a price that is hard to refuse -- there's no incentive to waste time and money reverse engineering anything. Reverse engineering is expensive. Make your product slightly less expensive.
Offer upgrades and enhancements that make any reverse engineering a bad idea. When the next release breaks their reverse engineering, there's no point. This can be carried to absurd extremes, but you should offer new features that make the next release more valuable than reverse engineering.
Offer customization at rates so attractive that they'd rather pay you to build and support the enhancements.
Use a license key which expires. This is cruel, and will give you a bad reputation, but it certainly makes your software stop working.
Offer it as a web service. SaaS involves no downloads to customers.
Python, being a byte-code-compiled interpreted language, is very difficult to lock down. Even if you use a exe-packager like py2exe, the layout of the executable is well-known, and the Python byte-codes are well understood.
Usually in cases like this, you have to make a tradeoff. How important is it really to protect the code? Are there real secrets in there (such as a key for symmetric encryption of bank transfers), or are you just being paranoid? Choose the language that lets you develop the best product quickest, and be realistic about how valuable your novel ideas are.
If you decide you really need to enforce the license check securely, write it as a small C extension so that the license check code can be extra-hard (but not impossible!) to reverse engineer, and leave the bulk of your code in Python.
Python is not the tool you need
You must use the right tool to do the right thing, and Python was not designed to be obfuscated. It's the contrary; everything is open or easy to reveal or modify in Python because that's the language's philosophy.
If you want something you can't see through, look for another tool. This is not a bad thing, it is important that several different tools exist for different usages.
Obfuscation is really hard
Even compiled programs can be reverse-engineered so don't think that you can fully protect any code. You can analyze obfuscated PHP, break the flash encryption key, etc. Newer versions of Windows are cracked every time.
Having a legal requirement is a good way to go
You cannot prevent somebody from misusing your code, but you can easily discover if someone does. Therefore, it's just a casual legal issue.
Code protection is overrated
Nowadays, business models tend to go for selling services instead of products. You cannot copy a service, pirate nor steal it. Maybe it's time to consider to go with the flow...
Compile python and distribute binaries!
Sensible idea:
Use Cython, Nuitka, Shed Skin or something similar to compile python to C code, then distribute your app as python binary libraries (pyd) instead.
That way, no Python (byte) code is left and you've done any reasonable amount of obscurification anyone (i.e. your employer) could expect from regular Code, I think. (.NET or Java less safe than this case, as that bytecode is not obfuscated and can relatively easily be decompiled into reasonable source.)
Cython is getting more and more compatible with CPython, so I think it should work. (I'm actually considering this for our product.. We're already building some thirdparty libs as pyd/dlls, so shipping our own python code as binaries is not a overly big step for us.)
See This Blog Post (not by me) for a tutorial on how to do it. (thx #hithwen)
Crazy idea:
You could probably get Cython to store the C-files separately for each module, then just concatenate them all and build them with heavy inlining. That way, your Python module is pretty monolithic and difficult to chip at with common tools.
Beyond crazy:
You might be able to build a single executable if you can link to (and optimize with) the python runtime and all libraries (dlls) statically. That way, it'd sure be difficult to intercept calls to/from python and whatever framework libraries you use. This cannot be done if you're using LGPL code though.
I understand that you want your customers to use the power of python but do not want expose the source code.
Here are my suggestions:
(a) Write the critical pieces of the code as C or C++ libraries and then use SIP or swig to expose the C/C++ APIs to Python namespace.
(b) Use cython instead of Python
(c) In both (a) and (b), it should be possible to distribute the libraries as licensed binary with a Python interface.
Have you had a look at pyminifier? It does Minify, obfuscate, and compress Python code. The example code looks pretty nasty for casual reverse engineering.
$ pyminifier --nonlatin --replacement-length=50 /tmp/tumult.py
#!/usr/bin/env python3
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲמּ=ImportError
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱=print
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ巡=False
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ澨=object
try:
import demiurgic
except ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲמּ:
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱("Warning: You're not demiurgic. Actually, I think that's normal.")
try:
import mystificate
except ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲמּ:
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱("Warning: Dark voodoo may be unreliable.")
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲﺬ=ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ巡
class ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐦚(ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ澨):
def __init__(self,*args,**kwargs):
pass
def ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ클(self,dactyl):
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ퐐=demiurgic.palpitation(dactyl)
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𠛲=mystificate.dark_voodoo(ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ퐐)
return ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𠛲
def ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐠯(self,whatever):
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱(whatever)
if __name__=="__main__":
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱("Forming...")
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲﺃ=ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐦚("epicaricacy","perseverate")
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲﺃ.ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐠯("Codswallop")
# Created by pyminifier (https://github.com/liftoff/pyminifier)
Is your employer aware that he can "steal" back any ideas that other people get from your code? I mean, if they can read your work, so can you theirs. Maybe looking at how you can benefit from the situation would yield a better return of your investment than fearing how much you could lose.
[EDIT] Answer to Nick's comment:
Nothing gained and nothing lost. The customer has what he wants (and paid for it since he did the change himself). Since he doesn't release the change, it's as if it didn't happen for everyone else.
Now if the customer sells the software, they have to change the copyright notice (which is illegal, so you can sue and will win -> simple case).
If they don't change the copyright notice, the 2nd level customers will notice that the software comes from you original and wonder what is going on. Chances are that they will contact you and so you will learn about the reselling of your work.
Again we have two cases: The original customer sold only a few copies. That means they didn't make much money anyway, so why bother. Or they sold in volume. That means better chances for you to learn about what they do and do something about it.
But in the end, most companies try to comply to the law (once their reputation is ruined, it's much harder to do business). So they will not steal your work but work with you to improve it. So if you include the source (with a license that protects you from simple reselling), chances are that they will simply push back changes they made since that will make sure the change is in the next version and they don't have to maintain it. That's win-win: You get changes and they can make the change themselves if they really, desperately need it even if you're unwilling to include it in the official release.
Use Cython. It will compile your modules to high-performant C files, which can then be compiled to native binary libraries. This is basically un-reversable, compared to .pyc bytecode!
I've written a detailed article on how to set up Cython for a Python project, check it out:
Protecting Python Sources With Cython
Do not rely on obfuscation. As You have correctly concluded, it offers very limited protection.
UPDATE: Here is a link to paper which reverse engineered obfuscated python code in Dropbox. The approach - opcode remapping is a good barrier, but clearly it can be defeated.
Instead, as many posters have mentioned make it:
Not worth reverse engineering time (Your software is so good, it makes sense to pay)
Make them sign a contract and do a license audit if feasible.
Alternatively, as the kick-ass Python IDE WingIDE does: Give away the code. That's right, give the code away and have people come back for upgrades and support.
Shipping .pyc files has its problems - they are not compatible with any other python version than the python version they were created with, which means you must know which python version is running on the systems the product will run on. That's a very limiting factor.
In some circumstances, it may be possible to move (all, or at least a key part) of the software into a web service that your organization hosts.
That way, the license checks can be performed in the safety of your own server room.
Though there's no perfect solution, the following can be done:
Move some critical piece of startup code into a native library.
Enforce the license check in the native library.
If the call to the native code were to be removed, the program wouldn't start anyway. If it's not removed then the license will be enforced.
Though this is not a cross-platform or a pure-Python solution, it will work.
I was surprised in not seeing pyconcrete in any answer. Maybe because it's newer than the question?
It could be exactly what you need(ed).
Instead of obfuscating the code, it encrypts it and decrypts at load time.
From pypi page:
Protect python script work flow
your_script.py import pyconcrete
pyconcrete will hook import module
when your script do import MODULE,
pyconcrete import hook will try to find MODULE.pye first and then
decrypt MODULE.pye via _pyconcrete.pyd and execute decrypted data (as
.pyc content)
encrypt & decrypt secret key record in _pyconcrete.pyd
(like DLL or SO) the secret key would be hide in binary code, can’t
see it directly in HEX view
The reliable only way to protect code is to run it on a server you control and provide your clients with a client which interfaces with that server.
I think there is one more method to protect your Python code; part of the Obfuscation method. I believe there was a game like Mount and Blade or something that changed and recompiled their own python interpreter (the original interpreter which i believe is open source) and just changed the OP codes in the OP code table to be different then the standard python OP codes.
So the python source is unmodified but the file extensions of the *.pyc files are different and the op codes don't match to the public python.exe interpreter. If you checked the games data files all the data was in Python source format.
All sorts of nasty tricks can be done to mess with immature hackers this way. Stopping a bunch of inexperienced hackers is easy. It's the professional hackers that you will not likely beat. But most companies don't keep pro hackers on staff long I imagine (likely because things get hacked). But immature hackers are all over the place (read as curious IT staff).
You could for example, in a modified interpreter, allow it to check for certain comments or doc strings in your source. You could have special OP codes for such lines of code. For example:
OP 234 is for source line "# Copyright I wrote this"
or compile that line into op codes that are equivalent to "if False:" if "# Copyright" is missing. Basically disabling a whole block of code for what appears to be some obscure reason.
One use case where recompiling a modified interpreter may be feasible is where you didn't write the app, the app is big, but you are paid to protect it, such as when you're a dedicated server admin for a financial app.
I find it a little contradictory to leave the source or opcodes open for eyeballs, but use SSL for network traffic. SSL is not 100% safe either. But it's used to stop MOST eyes from reading it. A wee bit precaution is sensible.
Also, if enough people deem that Python source and opcodes are too visible, it's likely someone will eventually develop at least a simple protection tool for it. So the more people asking "how to protect Python app" only promotes that development.
Depending in who the client is, a simple protection mechanism, combined with a sensible license agreement will be far more effective than any complex licensing/encryption/obfuscation system.
The best solution would be selling the code as a service, say by hosting the service, or offering support - although that isn't always practical.
Shipping the code as .pyc files will prevent your protection being foiled by a few #s, but it's hardly effective anti-piracy protection (as if there is such a technology), and at the end of the day, it shouldn't achieve anything that a decent license agreement with the company will.
Concentrate on making your code as nice to use as possible - having happy customers will make your company far more money than preventing some theoretical piracy..
Another attempt to make your code harder to steal is to use jython and then use java obfuscator.
This should work pretty well as jythonc translate python code to java and then java is compiled to bytecode. So ounce you obfuscate the classes it will be really hard to understand what is going on after decompilation, not to mention recovering the actual code.
The only problem with jython is that you can't use python modules written in c.
You should take a look at how the guys at getdropbox.com do it for their client software, including Linux. It's quite tricky to crack and requires some quite creative disassembly to get past the protection mechanisms.
The best you can do with Python is to obscure things.
Strip out all docstrings
Distribute only the .pyc compiled files.
freeze it
Obscure your constants inside a class/module so that help(config) doesn't show everything
You may be able to add some additional obscurity by encrypting part of it and decrypting it on the fly and passing it to eval(). But no matter what you do someone can break it.
None of this will stop a determined attacker from disassembling the bytecode or digging through your api with help, dir, etc.
What about signing your code with standard encryption schemes by hashing and signing important files and checking it with public key methods?
In this way you can issue license file with a public key for each customer.
Additional you can use an python obfuscator like this one (just googled it).
Idea of having time restricted license and check for it in locally installed program will not work. Even with perfect obfuscation, license check can be removed. However if you check license on remote system and run significant part of the program on your closed remote system, you will be able to protect your IP.
Preventing competitors from using the source code as their own or write their inspired version of the same code, one way to protect is to add signatures to your program logic (some secrets to be able to prove that code was stolen from you) and obfuscate the python source code so, it's hard to read and utilize.
Good obfuscation adds basically the same protection to your code, that compiling it to executable (and stripping binary) does. Figuring out how obfuscated complex code works might be even harder than actually writing your own implementation.
This will not help preventing hacking of your program. Even with obfuscation code license stuff will be cracked and program may be modified to have slightly different behaviour (in the same way that compiling code to binary does not help protection of native programs).
In addition to symbol obfuscation might be good idea to unrefactor the code, which makes everything even more confusing if e.g. call graphs points to many different places even if actually those different places does eventually the same thing.
Logical signature inside obfuscated code (e.g. you may create table of values which are used by program logic, but also used as signature), which can be used to determine that code is originated from you. If someone decides to use your obfuscated code module as part of their own product (even after reobfuscating it to make it seem different) you can show, that code is stolen with your secret signature.
I have looked at software protection in general for my own projects and the general philosophy is that complete protection is impossible. The only thing that you can hope to achieve is to add protection to a level that would cost your customer more to bypass than it would to purchase another license.
With that said I was just checking google for python obsfucation and not turning up a lot of anything. In a .Net solution, obsfucation would be a first approach to your problem on a windows platform, but I am not sure if anyone has solutions on Linux that work with Mono.
The next thing would be to write your code in a compiled language, or if you really want to go all the way, then in assembler. A stripped out executable would be a lot harder to decompile than an interpreted language.
It all comes down to tradeoffs. On one end you have ease of software development in python, in which it is also very hard to hide secrets. On the other end you have software written in assembler which is much harder to write, but is much easier to hide secrets.
Your boss has to choose a point somewhere along that continuum that supports his requirements. And then he has to give you the tools and time so you can build what he wants. However my bet is that he will object to real development costs versus potential monetary losses.
Neiher Cython nor Nuitka were not the answer, because when running the solution that is compiled with Nuitka or Cython into .pyd or .exe files a cache directory is generated and all .pyc files are copied into the cache directory, so an attacker simply can decompile .pyc files and see your code or change it.
It is possible to have the py2exe byte-code in a crypted resource for a C launcher that loads and executes it in memory. Some ideas here and here.
Some have also thought of a self modifying program to make reverse engineering expensive.
You can also find tutorials for preventing debuggers, make the disassembler fail, set false debugger breakpoints and protect your code with checksums. Search for ["crypted code" execute "in memory"] for more links.
But as others already said, if your code is worth it, reverse engineers will succeed in the end.
Use the same way to protect binary file of c/c++, that is, obfuscate each function body in executable or library binary file, insert an instruction "jump" at the begin of each function entry, jump to special function to restore obfuscated code. Byte-code is binary code of Python script, so
First compile python script to code object
Then iterate each code object, obfuscate co_code of each code object as the following
0 JUMP_ABSOLUTE n = 3 + len(bytecode)
3
...
... Here it's obfuscated bytecode
...
n LOAD_GLOBAL ? (__pyarmor__)
n+3 CALL_FUNCTION 0
n+6 POP_TOP
n+7 JUMP_ABSOLUTE 0
Save obfuscated code object as .pyc or .pyo file
Those obfuscated file (.pyc or .pyo) can be used by normal python interpreter, when those code object is called first time
First op is JUMP_ABSOLUTE, it will jump to offset n
At offset n, the instruction is to call a PyCFunction. This function will restore those obfuscated bytecode between offset 3 and n, and put the original byte-code at offset 0. The obfuscated code can be got by the following code
char *obfucated_bytecode;
Py_ssize_t len;
PyFrameObject* frame = PyEval_GetFrame();
PyCodeObject *f_code = frame->f_code;
PyObject *co_code = f_code->co_code;
PyBytes_AsStringAndSize(co_code, &obfucated_bytecode, &len)
After this function returns, the last instruction is to jump to
offset 0. The really byte-code now is executed.
There is a tool Pyarmor to obfuscate python scripts by this way.
There is a comprehensive answer on concealing the python source code, which can be find here.
Possible techniques discussed are:
- use compiled bytecode (python -m compileall)
- executable creators (or installers like PyInstaller)
- software as an service (the best solution to conceal your code in my opinion)
- python source code obfuscators
using cxfreeze ( py2exe for linux ) will do the job.
http://cx-freeze.sourceforge.net/
it is available in ubuntu repositories
If we focus on software licensing, I would recommend to take a look at another Stack Overflow answer I wrote here to get some inspiration of how a license key verification system can be constructed.
There is an open-source library on GitHub that can help you with the license verification bit.
You can install it by pip install licensing and then add the following code:
pubKey = "<RSAKeyValue><Modulus>sGbvxwdlDbqFXOMlVUnAF5ew0t0WpPW7rFpI5jHQOFkht/326dvh7t74RYeMpjy357NljouhpTLA3a6idnn4j6c3jmPWBkjZndGsPL4Bqm+fwE48nKpGPjkj4q/yzT4tHXBTyvaBjA8bVoCTnu+LiC4XEaLZRThGzIn5KQXKCigg6tQRy0GXE13XYFVz/x1mjFbT9/7dS8p85n8BuwlY5JvuBIQkKhuCNFfrUxBWyu87CFnXWjIupCD2VO/GbxaCvzrRjLZjAngLCMtZbYBALksqGPgTUN7ZM24XbPWyLtKPaXF2i4XRR9u6eTj5BfnLbKAU5PIVfjIS+vNYYogteQ==</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>"
res = Key.activate(token="WyIyNTU1IiwiRjdZZTB4RmtuTVcrQlNqcSszbmFMMHB3aWFJTlBsWW1Mbm9raVFyRyJd",\
rsa_pub_key=pubKey,\
product_id=3349, key="ICVLD-VVSZR-ZTICT-YKGXL", machine_code=Helpers.GetMachineCode())
if res[0] == None not Helpers.IsOnRightMachine(res[0]):
print("An error occured: {0}".format(res[1]))
else:
print("Success")
You can read more about the way the RSA public key, etc are configured here.
I documented how to obfuscate the python by converting it to .so file, and converting it to a python wheel file:
https://github.com/UM-NLP/python-obfuscation

Protect python code in raspberry pi [duplicate]

I am developing a piece of software in Python that will be distributed to my employer's customers. My employer wants to limit the usage of the software with a time-restricted license file.
If we distribute the .py files or even .pyc files it will be easy to (decompile and) remove the code that checks the license file.
Another aspect is that my employer does not want the code to be read by our customers, fearing that the code may be stolen or at least the "novel ideas".
Is there a good way to handle this problem?
"Is there a good way to handle this problem?" No. Nothing can be protected against reverse engineering. Even the firmware on DVD machines has been reverse engineered and the AACS Encryption key exposed. And that's in spite of the DMCA making that a criminal offense.
Since no technical method can stop your customers from reading your code, you have to apply ordinary commercial methods.
Licenses. Contracts. Terms and Conditions. This still works even when people can read the code. Note that some of your Python-based components may require that you pay fees before you sell software using those components. Also, some open-source licenses prohibit you from concealing the source or origins of that component.
Offer significant value. If your stuff is so good -- at a price that is hard to refuse -- there's no incentive to waste time and money reverse engineering anything. Reverse engineering is expensive. Make your product slightly less expensive.
Offer upgrades and enhancements that make any reverse engineering a bad idea. When the next release breaks their reverse engineering, there's no point. This can be carried to absurd extremes, but you should offer new features that make the next release more valuable than reverse engineering.
Offer customization at rates so attractive that they'd rather pay you to build and support the enhancements.
Use a license key which expires. This is cruel, and will give you a bad reputation, but it certainly makes your software stop working.
Offer it as a web service. SaaS involves no downloads to customers.
Python, being a byte-code-compiled interpreted language, is very difficult to lock down. Even if you use a exe-packager like py2exe, the layout of the executable is well-known, and the Python byte-codes are well understood.
Usually in cases like this, you have to make a tradeoff. How important is it really to protect the code? Are there real secrets in there (such as a key for symmetric encryption of bank transfers), or are you just being paranoid? Choose the language that lets you develop the best product quickest, and be realistic about how valuable your novel ideas are.
If you decide you really need to enforce the license check securely, write it as a small C extension so that the license check code can be extra-hard (but not impossible!) to reverse engineer, and leave the bulk of your code in Python.
Python is not the tool you need
You must use the right tool to do the right thing, and Python was not designed to be obfuscated. It's the contrary; everything is open or easy to reveal or modify in Python because that's the language's philosophy.
If you want something you can't see through, look for another tool. This is not a bad thing, it is important that several different tools exist for different usages.
Obfuscation is really hard
Even compiled programs can be reverse-engineered so don't think that you can fully protect any code. You can analyze obfuscated PHP, break the flash encryption key, etc. Newer versions of Windows are cracked every time.
Having a legal requirement is a good way to go
You cannot prevent somebody from misusing your code, but you can easily discover if someone does. Therefore, it's just a casual legal issue.
Code protection is overrated
Nowadays, business models tend to go for selling services instead of products. You cannot copy a service, pirate nor steal it. Maybe it's time to consider to go with the flow...
Compile python and distribute binaries!
Sensible idea:
Use Cython, Nuitka, Shed Skin or something similar to compile python to C code, then distribute your app as python binary libraries (pyd) instead.
That way, no Python (byte) code is left and you've done any reasonable amount of obscurification anyone (i.e. your employer) could expect from regular Code, I think. (.NET or Java less safe than this case, as that bytecode is not obfuscated and can relatively easily be decompiled into reasonable source.)
Cython is getting more and more compatible with CPython, so I think it should work. (I'm actually considering this for our product.. We're already building some thirdparty libs as pyd/dlls, so shipping our own python code as binaries is not a overly big step for us.)
See This Blog Post (not by me) for a tutorial on how to do it. (thx #hithwen)
Crazy idea:
You could probably get Cython to store the C-files separately for each module, then just concatenate them all and build them with heavy inlining. That way, your Python module is pretty monolithic and difficult to chip at with common tools.
Beyond crazy:
You might be able to build a single executable if you can link to (and optimize with) the python runtime and all libraries (dlls) statically. That way, it'd sure be difficult to intercept calls to/from python and whatever framework libraries you use. This cannot be done if you're using LGPL code though.
I understand that you want your customers to use the power of python but do not want expose the source code.
Here are my suggestions:
(a) Write the critical pieces of the code as C or C++ libraries and then use SIP or swig to expose the C/C++ APIs to Python namespace.
(b) Use cython instead of Python
(c) In both (a) and (b), it should be possible to distribute the libraries as licensed binary with a Python interface.
Have you had a look at pyminifier? It does Minify, obfuscate, and compress Python code. The example code looks pretty nasty for casual reverse engineering.
$ pyminifier --nonlatin --replacement-length=50 /tmp/tumult.py
#!/usr/bin/env python3
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲמּ=ImportError
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱=print
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ巡=False
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ澨=object
try:
import demiurgic
except ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲמּ:
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱("Warning: You're not demiurgic. Actually, I think that's normal.")
try:
import mystificate
except ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲמּ:
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱("Warning: Dark voodoo may be unreliable.")
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲﺬ=ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ巡
class ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐦚(ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ澨):
def __init__(self,*args,**kwargs):
pass
def ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ클(self,dactyl):
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ퐐=demiurgic.palpitation(dactyl)
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𠛲=mystificate.dark_voodoo(ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ퐐)
return ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𠛲
def ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐠯(self,whatever):
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱(whatever)
if __name__=="__main__":
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱("Forming...")
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲﺃ=ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐦚("epicaricacy","perseverate")
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲﺃ.ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐠯("Codswallop")
# Created by pyminifier (https://github.com/liftoff/pyminifier)
Is your employer aware that he can "steal" back any ideas that other people get from your code? I mean, if they can read your work, so can you theirs. Maybe looking at how you can benefit from the situation would yield a better return of your investment than fearing how much you could lose.
[EDIT] Answer to Nick's comment:
Nothing gained and nothing lost. The customer has what he wants (and paid for it since he did the change himself). Since he doesn't release the change, it's as if it didn't happen for everyone else.
Now if the customer sells the software, they have to change the copyright notice (which is illegal, so you can sue and will win -> simple case).
If they don't change the copyright notice, the 2nd level customers will notice that the software comes from you original and wonder what is going on. Chances are that they will contact you and so you will learn about the reselling of your work.
Again we have two cases: The original customer sold only a few copies. That means they didn't make much money anyway, so why bother. Or they sold in volume. That means better chances for you to learn about what they do and do something about it.
But in the end, most companies try to comply to the law (once their reputation is ruined, it's much harder to do business). So they will not steal your work but work with you to improve it. So if you include the source (with a license that protects you from simple reselling), chances are that they will simply push back changes they made since that will make sure the change is in the next version and they don't have to maintain it. That's win-win: You get changes and they can make the change themselves if they really, desperately need it even if you're unwilling to include it in the official release.
Use Cython. It will compile your modules to high-performant C files, which can then be compiled to native binary libraries. This is basically un-reversable, compared to .pyc bytecode!
I've written a detailed article on how to set up Cython for a Python project, check it out:
Protecting Python Sources With Cython
Do not rely on obfuscation. As You have correctly concluded, it offers very limited protection.
UPDATE: Here is a link to paper which reverse engineered obfuscated python code in Dropbox. The approach - opcode remapping is a good barrier, but clearly it can be defeated.
Instead, as many posters have mentioned make it:
Not worth reverse engineering time (Your software is so good, it makes sense to pay)
Make them sign a contract and do a license audit if feasible.
Alternatively, as the kick-ass Python IDE WingIDE does: Give away the code. That's right, give the code away and have people come back for upgrades and support.
Shipping .pyc files has its problems - they are not compatible with any other python version than the python version they were created with, which means you must know which python version is running on the systems the product will run on. That's a very limiting factor.
In some circumstances, it may be possible to move (all, or at least a key part) of the software into a web service that your organization hosts.
That way, the license checks can be performed in the safety of your own server room.
Though there's no perfect solution, the following can be done:
Move some critical piece of startup code into a native library.
Enforce the license check in the native library.
If the call to the native code were to be removed, the program wouldn't start anyway. If it's not removed then the license will be enforced.
Though this is not a cross-platform or a pure-Python solution, it will work.
I was surprised in not seeing pyconcrete in any answer. Maybe because it's newer than the question?
It could be exactly what you need(ed).
Instead of obfuscating the code, it encrypts it and decrypts at load time.
From pypi page:
Protect python script work flow
your_script.py import pyconcrete
pyconcrete will hook import module
when your script do import MODULE,
pyconcrete import hook will try to find MODULE.pye first and then
decrypt MODULE.pye via _pyconcrete.pyd and execute decrypted data (as
.pyc content)
encrypt & decrypt secret key record in _pyconcrete.pyd
(like DLL or SO) the secret key would be hide in binary code, can’t
see it directly in HEX view
The reliable only way to protect code is to run it on a server you control and provide your clients with a client which interfaces with that server.
I think there is one more method to protect your Python code; part of the Obfuscation method. I believe there was a game like Mount and Blade or something that changed and recompiled their own python interpreter (the original interpreter which i believe is open source) and just changed the OP codes in the OP code table to be different then the standard python OP codes.
So the python source is unmodified but the file extensions of the *.pyc files are different and the op codes don't match to the public python.exe interpreter. If you checked the games data files all the data was in Python source format.
All sorts of nasty tricks can be done to mess with immature hackers this way. Stopping a bunch of inexperienced hackers is easy. It's the professional hackers that you will not likely beat. But most companies don't keep pro hackers on staff long I imagine (likely because things get hacked). But immature hackers are all over the place (read as curious IT staff).
You could for example, in a modified interpreter, allow it to check for certain comments or doc strings in your source. You could have special OP codes for such lines of code. For example:
OP 234 is for source line "# Copyright I wrote this"
or compile that line into op codes that are equivalent to "if False:" if "# Copyright" is missing. Basically disabling a whole block of code for what appears to be some obscure reason.
One use case where recompiling a modified interpreter may be feasible is where you didn't write the app, the app is big, but you are paid to protect it, such as when you're a dedicated server admin for a financial app.
I find it a little contradictory to leave the source or opcodes open for eyeballs, but use SSL for network traffic. SSL is not 100% safe either. But it's used to stop MOST eyes from reading it. A wee bit precaution is sensible.
Also, if enough people deem that Python source and opcodes are too visible, it's likely someone will eventually develop at least a simple protection tool for it. So the more people asking "how to protect Python app" only promotes that development.
Depending in who the client is, a simple protection mechanism, combined with a sensible license agreement will be far more effective than any complex licensing/encryption/obfuscation system.
The best solution would be selling the code as a service, say by hosting the service, or offering support - although that isn't always practical.
Shipping the code as .pyc files will prevent your protection being foiled by a few #s, but it's hardly effective anti-piracy protection (as if there is such a technology), and at the end of the day, it shouldn't achieve anything that a decent license agreement with the company will.
Concentrate on making your code as nice to use as possible - having happy customers will make your company far more money than preventing some theoretical piracy..
Another attempt to make your code harder to steal is to use jython and then use java obfuscator.
This should work pretty well as jythonc translate python code to java and then java is compiled to bytecode. So ounce you obfuscate the classes it will be really hard to understand what is going on after decompilation, not to mention recovering the actual code.
The only problem with jython is that you can't use python modules written in c.
You should take a look at how the guys at getdropbox.com do it for their client software, including Linux. It's quite tricky to crack and requires some quite creative disassembly to get past the protection mechanisms.
The best you can do with Python is to obscure things.
Strip out all docstrings
Distribute only the .pyc compiled files.
freeze it
Obscure your constants inside a class/module so that help(config) doesn't show everything
You may be able to add some additional obscurity by encrypting part of it and decrypting it on the fly and passing it to eval(). But no matter what you do someone can break it.
None of this will stop a determined attacker from disassembling the bytecode or digging through your api with help, dir, etc.
What about signing your code with standard encryption schemes by hashing and signing important files and checking it with public key methods?
In this way you can issue license file with a public key for each customer.
Additional you can use an python obfuscator like this one (just googled it).
Idea of having time restricted license and check for it in locally installed program will not work. Even with perfect obfuscation, license check can be removed. However if you check license on remote system and run significant part of the program on your closed remote system, you will be able to protect your IP.
Preventing competitors from using the source code as their own or write their inspired version of the same code, one way to protect is to add signatures to your program logic (some secrets to be able to prove that code was stolen from you) and obfuscate the python source code so, it's hard to read and utilize.
Good obfuscation adds basically the same protection to your code, that compiling it to executable (and stripping binary) does. Figuring out how obfuscated complex code works might be even harder than actually writing your own implementation.
This will not help preventing hacking of your program. Even with obfuscation code license stuff will be cracked and program may be modified to have slightly different behaviour (in the same way that compiling code to binary does not help protection of native programs).
In addition to symbol obfuscation might be good idea to unrefactor the code, which makes everything even more confusing if e.g. call graphs points to many different places even if actually those different places does eventually the same thing.
Logical signature inside obfuscated code (e.g. you may create table of values which are used by program logic, but also used as signature), which can be used to determine that code is originated from you. If someone decides to use your obfuscated code module as part of their own product (even after reobfuscating it to make it seem different) you can show, that code is stolen with your secret signature.
I have looked at software protection in general for my own projects and the general philosophy is that complete protection is impossible. The only thing that you can hope to achieve is to add protection to a level that would cost your customer more to bypass than it would to purchase another license.
With that said I was just checking google for python obsfucation and not turning up a lot of anything. In a .Net solution, obsfucation would be a first approach to your problem on a windows platform, but I am not sure if anyone has solutions on Linux that work with Mono.
The next thing would be to write your code in a compiled language, or if you really want to go all the way, then in assembler. A stripped out executable would be a lot harder to decompile than an interpreted language.
It all comes down to tradeoffs. On one end you have ease of software development in python, in which it is also very hard to hide secrets. On the other end you have software written in assembler which is much harder to write, but is much easier to hide secrets.
Your boss has to choose a point somewhere along that continuum that supports his requirements. And then he has to give you the tools and time so you can build what he wants. However my bet is that he will object to real development costs versus potential monetary losses.
Neiher Cython nor Nuitka were not the answer, because when running the solution that is compiled with Nuitka or Cython into .pyd or .exe files a cache directory is generated and all .pyc files are copied into the cache directory, so an attacker simply can decompile .pyc files and see your code or change it.
It is possible to have the py2exe byte-code in a crypted resource for a C launcher that loads and executes it in memory. Some ideas here and here.
Some have also thought of a self modifying program to make reverse engineering expensive.
You can also find tutorials for preventing debuggers, make the disassembler fail, set false debugger breakpoints and protect your code with checksums. Search for ["crypted code" execute "in memory"] for more links.
But as others already said, if your code is worth it, reverse engineers will succeed in the end.
Use the same way to protect binary file of c/c++, that is, obfuscate each function body in executable or library binary file, insert an instruction "jump" at the begin of each function entry, jump to special function to restore obfuscated code. Byte-code is binary code of Python script, so
First compile python script to code object
Then iterate each code object, obfuscate co_code of each code object as the following
0 JUMP_ABSOLUTE n = 3 + len(bytecode)
3
...
... Here it's obfuscated bytecode
...
n LOAD_GLOBAL ? (__pyarmor__)
n+3 CALL_FUNCTION 0
n+6 POP_TOP
n+7 JUMP_ABSOLUTE 0
Save obfuscated code object as .pyc or .pyo file
Those obfuscated file (.pyc or .pyo) can be used by normal python interpreter, when those code object is called first time
First op is JUMP_ABSOLUTE, it will jump to offset n
At offset n, the instruction is to call a PyCFunction. This function will restore those obfuscated bytecode between offset 3 and n, and put the original byte-code at offset 0. The obfuscated code can be got by the following code
char *obfucated_bytecode;
Py_ssize_t len;
PyFrameObject* frame = PyEval_GetFrame();
PyCodeObject *f_code = frame->f_code;
PyObject *co_code = f_code->co_code;
PyBytes_AsStringAndSize(co_code, &obfucated_bytecode, &len)
After this function returns, the last instruction is to jump to
offset 0. The really byte-code now is executed.
There is a tool Pyarmor to obfuscate python scripts by this way.
There is a comprehensive answer on concealing the python source code, which can be find here.
Possible techniques discussed are:
- use compiled bytecode (python -m compileall)
- executable creators (or installers like PyInstaller)
- software as an service (the best solution to conceal your code in my opinion)
- python source code obfuscators
using cxfreeze ( py2exe for linux ) will do the job.
http://cx-freeze.sourceforge.net/
it is available in ubuntu repositories
If we focus on software licensing, I would recommend to take a look at another Stack Overflow answer I wrote here to get some inspiration of how a license key verification system can be constructed.
There is an open-source library on GitHub that can help you with the license verification bit.
You can install it by pip install licensing and then add the following code:
pubKey = "<RSAKeyValue><Modulus>sGbvxwdlDbqFXOMlVUnAF5ew0t0WpPW7rFpI5jHQOFkht/326dvh7t74RYeMpjy357NljouhpTLA3a6idnn4j6c3jmPWBkjZndGsPL4Bqm+fwE48nKpGPjkj4q/yzT4tHXBTyvaBjA8bVoCTnu+LiC4XEaLZRThGzIn5KQXKCigg6tQRy0GXE13XYFVz/x1mjFbT9/7dS8p85n8BuwlY5JvuBIQkKhuCNFfrUxBWyu87CFnXWjIupCD2VO/GbxaCvzrRjLZjAngLCMtZbYBALksqGPgTUN7ZM24XbPWyLtKPaXF2i4XRR9u6eTj5BfnLbKAU5PIVfjIS+vNYYogteQ==</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>"
res = Key.activate(token="WyIyNTU1IiwiRjdZZTB4RmtuTVcrQlNqcSszbmFMMHB3aWFJTlBsWW1Mbm9raVFyRyJd",\
rsa_pub_key=pubKey,\
product_id=3349, key="ICVLD-VVSZR-ZTICT-YKGXL", machine_code=Helpers.GetMachineCode())
if res[0] == None not Helpers.IsOnRightMachine(res[0]):
print("An error occured: {0}".format(res[1]))
else:
print("Success")
You can read more about the way the RSA public key, etc are configured here.
I documented how to obfuscate the python by converting it to .so file, and converting it to a python wheel file:
https://github.com/UM-NLP/python-obfuscation

Is it possible to compile Python source codes to protect code from unauthorized access? [duplicate]

I am developing a piece of software in Python that will be distributed to my employer's customers. My employer wants to limit the usage of the software with a time-restricted license file.
If we distribute the .py files or even .pyc files it will be easy to (decompile and) remove the code that checks the license file.
Another aspect is that my employer does not want the code to be read by our customers, fearing that the code may be stolen or at least the "novel ideas".
Is there a good way to handle this problem?
"Is there a good way to handle this problem?" No. Nothing can be protected against reverse engineering. Even the firmware on DVD machines has been reverse engineered and the AACS Encryption key exposed. And that's in spite of the DMCA making that a criminal offense.
Since no technical method can stop your customers from reading your code, you have to apply ordinary commercial methods.
Licenses. Contracts. Terms and Conditions. This still works even when people can read the code. Note that some of your Python-based components may require that you pay fees before you sell software using those components. Also, some open-source licenses prohibit you from concealing the source or origins of that component.
Offer significant value. If your stuff is so good -- at a price that is hard to refuse -- there's no incentive to waste time and money reverse engineering anything. Reverse engineering is expensive. Make your product slightly less expensive.
Offer upgrades and enhancements that make any reverse engineering a bad idea. When the next release breaks their reverse engineering, there's no point. This can be carried to absurd extremes, but you should offer new features that make the next release more valuable than reverse engineering.
Offer customization at rates so attractive that they'd rather pay you to build and support the enhancements.
Use a license key which expires. This is cruel, and will give you a bad reputation, but it certainly makes your software stop working.
Offer it as a web service. SaaS involves no downloads to customers.
Python, being a byte-code-compiled interpreted language, is very difficult to lock down. Even if you use a exe-packager like py2exe, the layout of the executable is well-known, and the Python byte-codes are well understood.
Usually in cases like this, you have to make a tradeoff. How important is it really to protect the code? Are there real secrets in there (such as a key for symmetric encryption of bank transfers), or are you just being paranoid? Choose the language that lets you develop the best product quickest, and be realistic about how valuable your novel ideas are.
If you decide you really need to enforce the license check securely, write it as a small C extension so that the license check code can be extra-hard (but not impossible!) to reverse engineer, and leave the bulk of your code in Python.
Python is not the tool you need
You must use the right tool to do the right thing, and Python was not designed to be obfuscated. It's the contrary; everything is open or easy to reveal or modify in Python because that's the language's philosophy.
If you want something you can't see through, look for another tool. This is not a bad thing, it is important that several different tools exist for different usages.
Obfuscation is really hard
Even compiled programs can be reverse-engineered so don't think that you can fully protect any code. You can analyze obfuscated PHP, break the flash encryption key, etc. Newer versions of Windows are cracked every time.
Having a legal requirement is a good way to go
You cannot prevent somebody from misusing your code, but you can easily discover if someone does. Therefore, it's just a casual legal issue.
Code protection is overrated
Nowadays, business models tend to go for selling services instead of products. You cannot copy a service, pirate nor steal it. Maybe it's time to consider to go with the flow...
Compile python and distribute binaries!
Sensible idea:
Use Cython, Nuitka, Shed Skin or something similar to compile python to C code, then distribute your app as python binary libraries (pyd) instead.
That way, no Python (byte) code is left and you've done any reasonable amount of obscurification anyone (i.e. your employer) could expect from regular Code, I think. (.NET or Java less safe than this case, as that bytecode is not obfuscated and can relatively easily be decompiled into reasonable source.)
Cython is getting more and more compatible with CPython, so I think it should work. (I'm actually considering this for our product.. We're already building some thirdparty libs as pyd/dlls, so shipping our own python code as binaries is not a overly big step for us.)
See This Blog Post (not by me) for a tutorial on how to do it. (thx #hithwen)
Crazy idea:
You could probably get Cython to store the C-files separately for each module, then just concatenate them all and build them with heavy inlining. That way, your Python module is pretty monolithic and difficult to chip at with common tools.
Beyond crazy:
You might be able to build a single executable if you can link to (and optimize with) the python runtime and all libraries (dlls) statically. That way, it'd sure be difficult to intercept calls to/from python and whatever framework libraries you use. This cannot be done if you're using LGPL code though.
I understand that you want your customers to use the power of python but do not want expose the source code.
Here are my suggestions:
(a) Write the critical pieces of the code as C or C++ libraries and then use SIP or swig to expose the C/C++ APIs to Python namespace.
(b) Use cython instead of Python
(c) In both (a) and (b), it should be possible to distribute the libraries as licensed binary with a Python interface.
Have you had a look at pyminifier? It does Minify, obfuscate, and compress Python code. The example code looks pretty nasty for casual reverse engineering.
$ pyminifier --nonlatin --replacement-length=50 /tmp/tumult.py
#!/usr/bin/env python3
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲמּ=ImportError
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱=print
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ巡=False
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ澨=object
try:
import demiurgic
except ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲמּ:
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱("Warning: You're not demiurgic. Actually, I think that's normal.")
try:
import mystificate
except ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲמּ:
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱("Warning: Dark voodoo may be unreliable.")
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲﺬ=ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ巡
class ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐦚(ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ澨):
def __init__(self,*args,**kwargs):
pass
def ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ클(self,dactyl):
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ퐐=demiurgic.palpitation(dactyl)
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𠛲=mystificate.dark_voodoo(ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ퐐)
return ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𠛲
def ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐠯(self,whatever):
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱(whatever)
if __name__=="__main__":
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ燱("Forming...")
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲﺃ=ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐦚("epicaricacy","perseverate")
ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲﺃ.ﺭ异𞸐𐤔ﭞﰣﺁں𝕌𨿩𞸇뻛𐬑𥰫嬭ﱌ𢽁𐡆𧪮Ꝫﴹ뙫𢤴퉊ﳦﲣפּܟﺶ𐐤ﶨࠔ𐰷𢡶𧐎𐭈𞸏𢢘𦘼ﶻ𩏃𦽨𞺎𠛘𐠲䉊ﰸﭳᣲ𐠯("Codswallop")
# Created by pyminifier (https://github.com/liftoff/pyminifier)
Is your employer aware that he can "steal" back any ideas that other people get from your code? I mean, if they can read your work, so can you theirs. Maybe looking at how you can benefit from the situation would yield a better return of your investment than fearing how much you could lose.
[EDIT] Answer to Nick's comment:
Nothing gained and nothing lost. The customer has what he wants (and paid for it since he did the change himself). Since he doesn't release the change, it's as if it didn't happen for everyone else.
Now if the customer sells the software, they have to change the copyright notice (which is illegal, so you can sue and will win -> simple case).
If they don't change the copyright notice, the 2nd level customers will notice that the software comes from you original and wonder what is going on. Chances are that they will contact you and so you will learn about the reselling of your work.
Again we have two cases: The original customer sold only a few copies. That means they didn't make much money anyway, so why bother. Or they sold in volume. That means better chances for you to learn about what they do and do something about it.
But in the end, most companies try to comply to the law (once their reputation is ruined, it's much harder to do business). So they will not steal your work but work with you to improve it. So if you include the source (with a license that protects you from simple reselling), chances are that they will simply push back changes they made since that will make sure the change is in the next version and they don't have to maintain it. That's win-win: You get changes and they can make the change themselves if they really, desperately need it even if you're unwilling to include it in the official release.
Use Cython. It will compile your modules to high-performant C files, which can then be compiled to native binary libraries. This is basically un-reversable, compared to .pyc bytecode!
I've written a detailed article on how to set up Cython for a Python project, check it out:
Protecting Python Sources With Cython
Do not rely on obfuscation. As You have correctly concluded, it offers very limited protection.
UPDATE: Here is a link to paper which reverse engineered obfuscated python code in Dropbox. The approach - opcode remapping is a good barrier, but clearly it can be defeated.
Instead, as many posters have mentioned make it:
Not worth reverse engineering time (Your software is so good, it makes sense to pay)
Make them sign a contract and do a license audit if feasible.
Alternatively, as the kick-ass Python IDE WingIDE does: Give away the code. That's right, give the code away and have people come back for upgrades and support.
Shipping .pyc files has its problems - they are not compatible with any other python version than the python version they were created with, which means you must know which python version is running on the systems the product will run on. That's a very limiting factor.
In some circumstances, it may be possible to move (all, or at least a key part) of the software into a web service that your organization hosts.
That way, the license checks can be performed in the safety of your own server room.
Though there's no perfect solution, the following can be done:
Move some critical piece of startup code into a native library.
Enforce the license check in the native library.
If the call to the native code were to be removed, the program wouldn't start anyway. If it's not removed then the license will be enforced.
Though this is not a cross-platform or a pure-Python solution, it will work.
I was surprised in not seeing pyconcrete in any answer. Maybe because it's newer than the question?
It could be exactly what you need(ed).
Instead of obfuscating the code, it encrypts it and decrypts at load time.
From pypi page:
Protect python script work flow
your_script.py import pyconcrete
pyconcrete will hook import module
when your script do import MODULE,
pyconcrete import hook will try to find MODULE.pye first and then
decrypt MODULE.pye via _pyconcrete.pyd and execute decrypted data (as
.pyc content)
encrypt & decrypt secret key record in _pyconcrete.pyd
(like DLL or SO) the secret key would be hide in binary code, can’t
see it directly in HEX view
The reliable only way to protect code is to run it on a server you control and provide your clients with a client which interfaces with that server.
I think there is one more method to protect your Python code; part of the Obfuscation method. I believe there was a game like Mount and Blade or something that changed and recompiled their own python interpreter (the original interpreter which i believe is open source) and just changed the OP codes in the OP code table to be different then the standard python OP codes.
So the python source is unmodified but the file extensions of the *.pyc files are different and the op codes don't match to the public python.exe interpreter. If you checked the games data files all the data was in Python source format.
All sorts of nasty tricks can be done to mess with immature hackers this way. Stopping a bunch of inexperienced hackers is easy. It's the professional hackers that you will not likely beat. But most companies don't keep pro hackers on staff long I imagine (likely because things get hacked). But immature hackers are all over the place (read as curious IT staff).
You could for example, in a modified interpreter, allow it to check for certain comments or doc strings in your source. You could have special OP codes for such lines of code. For example:
OP 234 is for source line "# Copyright I wrote this"
or compile that line into op codes that are equivalent to "if False:" if "# Copyright" is missing. Basically disabling a whole block of code for what appears to be some obscure reason.
One use case where recompiling a modified interpreter may be feasible is where you didn't write the app, the app is big, but you are paid to protect it, such as when you're a dedicated server admin for a financial app.
I find it a little contradictory to leave the source or opcodes open for eyeballs, but use SSL for network traffic. SSL is not 100% safe either. But it's used to stop MOST eyes from reading it. A wee bit precaution is sensible.
Also, if enough people deem that Python source and opcodes are too visible, it's likely someone will eventually develop at least a simple protection tool for it. So the more people asking "how to protect Python app" only promotes that development.
Depending in who the client is, a simple protection mechanism, combined with a sensible license agreement will be far more effective than any complex licensing/encryption/obfuscation system.
The best solution would be selling the code as a service, say by hosting the service, or offering support - although that isn't always practical.
Shipping the code as .pyc files will prevent your protection being foiled by a few #s, but it's hardly effective anti-piracy protection (as if there is such a technology), and at the end of the day, it shouldn't achieve anything that a decent license agreement with the company will.
Concentrate on making your code as nice to use as possible - having happy customers will make your company far more money than preventing some theoretical piracy..
Another attempt to make your code harder to steal is to use jython and then use java obfuscator.
This should work pretty well as jythonc translate python code to java and then java is compiled to bytecode. So ounce you obfuscate the classes it will be really hard to understand what is going on after decompilation, not to mention recovering the actual code.
The only problem with jython is that you can't use python modules written in c.
You should take a look at how the guys at getdropbox.com do it for their client software, including Linux. It's quite tricky to crack and requires some quite creative disassembly to get past the protection mechanisms.
The best you can do with Python is to obscure things.
Strip out all docstrings
Distribute only the .pyc compiled files.
freeze it
Obscure your constants inside a class/module so that help(config) doesn't show everything
You may be able to add some additional obscurity by encrypting part of it and decrypting it on the fly and passing it to eval(). But no matter what you do someone can break it.
None of this will stop a determined attacker from disassembling the bytecode or digging through your api with help, dir, etc.
What about signing your code with standard encryption schemes by hashing and signing important files and checking it with public key methods?
In this way you can issue license file with a public key for each customer.
Additional you can use an python obfuscator like this one (just googled it).
Idea of having time restricted license and check for it in locally installed program will not work. Even with perfect obfuscation, license check can be removed. However if you check license on remote system and run significant part of the program on your closed remote system, you will be able to protect your IP.
Preventing competitors from using the source code as their own or write their inspired version of the same code, one way to protect is to add signatures to your program logic (some secrets to be able to prove that code was stolen from you) and obfuscate the python source code so, it's hard to read and utilize.
Good obfuscation adds basically the same protection to your code, that compiling it to executable (and stripping binary) does. Figuring out how obfuscated complex code works might be even harder than actually writing your own implementation.
This will not help preventing hacking of your program. Even with obfuscation code license stuff will be cracked and program may be modified to have slightly different behaviour (in the same way that compiling code to binary does not help protection of native programs).
In addition to symbol obfuscation might be good idea to unrefactor the code, which makes everything even more confusing if e.g. call graphs points to many different places even if actually those different places does eventually the same thing.
Logical signature inside obfuscated code (e.g. you may create table of values which are used by program logic, but also used as signature), which can be used to determine that code is originated from you. If someone decides to use your obfuscated code module as part of their own product (even after reobfuscating it to make it seem different) you can show, that code is stolen with your secret signature.
I have looked at software protection in general for my own projects and the general philosophy is that complete protection is impossible. The only thing that you can hope to achieve is to add protection to a level that would cost your customer more to bypass than it would to purchase another license.
With that said I was just checking google for python obsfucation and not turning up a lot of anything. In a .Net solution, obsfucation would be a first approach to your problem on a windows platform, but I am not sure if anyone has solutions on Linux that work with Mono.
The next thing would be to write your code in a compiled language, or if you really want to go all the way, then in assembler. A stripped out executable would be a lot harder to decompile than an interpreted language.
It all comes down to tradeoffs. On one end you have ease of software development in python, in which it is also very hard to hide secrets. On the other end you have software written in assembler which is much harder to write, but is much easier to hide secrets.
Your boss has to choose a point somewhere along that continuum that supports his requirements. And then he has to give you the tools and time so you can build what he wants. However my bet is that he will object to real development costs versus potential monetary losses.
Neiher Cython nor Nuitka were not the answer, because when running the solution that is compiled with Nuitka or Cython into .pyd or .exe files a cache directory is generated and all .pyc files are copied into the cache directory, so an attacker simply can decompile .pyc files and see your code or change it.
It is possible to have the py2exe byte-code in a crypted resource for a C launcher that loads and executes it in memory. Some ideas here and here.
Some have also thought of a self modifying program to make reverse engineering expensive.
You can also find tutorials for preventing debuggers, make the disassembler fail, set false debugger breakpoints and protect your code with checksums. Search for ["crypted code" execute "in memory"] for more links.
But as others already said, if your code is worth it, reverse engineers will succeed in the end.
Use the same way to protect binary file of c/c++, that is, obfuscate each function body in executable or library binary file, insert an instruction "jump" at the begin of each function entry, jump to special function to restore obfuscated code. Byte-code is binary code of Python script, so
First compile python script to code object
Then iterate each code object, obfuscate co_code of each code object as the following
0 JUMP_ABSOLUTE n = 3 + len(bytecode)
3
...
... Here it's obfuscated bytecode
...
n LOAD_GLOBAL ? (__pyarmor__)
n+3 CALL_FUNCTION 0
n+6 POP_TOP
n+7 JUMP_ABSOLUTE 0
Save obfuscated code object as .pyc or .pyo file
Those obfuscated file (.pyc or .pyo) can be used by normal python interpreter, when those code object is called first time
First op is JUMP_ABSOLUTE, it will jump to offset n
At offset n, the instruction is to call a PyCFunction. This function will restore those obfuscated bytecode between offset 3 and n, and put the original byte-code at offset 0. The obfuscated code can be got by the following code
char *obfucated_bytecode;
Py_ssize_t len;
PyFrameObject* frame = PyEval_GetFrame();
PyCodeObject *f_code = frame->f_code;
PyObject *co_code = f_code->co_code;
PyBytes_AsStringAndSize(co_code, &obfucated_bytecode, &len)
After this function returns, the last instruction is to jump to
offset 0. The really byte-code now is executed.
There is a tool Pyarmor to obfuscate python scripts by this way.
There is a comprehensive answer on concealing the python source code, which can be find here.
Possible techniques discussed are:
- use compiled bytecode (python -m compileall)
- executable creators (or installers like PyInstaller)
- software as an service (the best solution to conceal your code in my opinion)
- python source code obfuscators
using cxfreeze ( py2exe for linux ) will do the job.
http://cx-freeze.sourceforge.net/
it is available in ubuntu repositories
If we focus on software licensing, I would recommend to take a look at another Stack Overflow answer I wrote here to get some inspiration of how a license key verification system can be constructed.
There is an open-source library on GitHub that can help you with the license verification bit.
You can install it by pip install licensing and then add the following code:
pubKey = "<RSAKeyValue><Modulus>sGbvxwdlDbqFXOMlVUnAF5ew0t0WpPW7rFpI5jHQOFkht/326dvh7t74RYeMpjy357NljouhpTLA3a6idnn4j6c3jmPWBkjZndGsPL4Bqm+fwE48nKpGPjkj4q/yzT4tHXBTyvaBjA8bVoCTnu+LiC4XEaLZRThGzIn5KQXKCigg6tQRy0GXE13XYFVz/x1mjFbT9/7dS8p85n8BuwlY5JvuBIQkKhuCNFfrUxBWyu87CFnXWjIupCD2VO/GbxaCvzrRjLZjAngLCMtZbYBALksqGPgTUN7ZM24XbPWyLtKPaXF2i4XRR9u6eTj5BfnLbKAU5PIVfjIS+vNYYogteQ==</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>"
res = Key.activate(token="WyIyNTU1IiwiRjdZZTB4RmtuTVcrQlNqcSszbmFMMHB3aWFJTlBsWW1Mbm9raVFyRyJd",\
rsa_pub_key=pubKey,\
product_id=3349, key="ICVLD-VVSZR-ZTICT-YKGXL", machine_code=Helpers.GetMachineCode())
if res[0] == None not Helpers.IsOnRightMachine(res[0]):
print("An error occured: {0}".format(res[1]))
else:
print("Success")
You can read more about the way the RSA public key, etc are configured here.
I documented how to obfuscate the python by converting it to .so file, and converting it to a python wheel file:
https://github.com/UM-NLP/python-obfuscation

Identifying "sensitive" code in your application

Looking to improve quality of a fairly large Python project. I am happy with the types of warnings PyLint gives me. However, they are just too numerous and hard to enforce across a large organization. Also I believe that some code is more critical/sensitive than others with respect to where the next bug may come. For example I would like to spend more time validating a library method that is used by 100 modules rather than a script that was last touched 2 years ago and may not be used in production. Also it would be interesting to know modules that are frequently updated.
Is anyone familiar with tools for Python or otherwise that help with this type of analysis?
You problem is similar to the one I answered over at SQA https://sqa.stackexchange.com/a/3082. This problem was associated with Java which made the tooling a bit easier, but I have a number of suggestions below.
A number of other answers suggest that there is no good runtime tools for Python. I disagree on this in several ways:
Coverage tools work very well
Based on my experience in tooling in Java, static and dynamic analysis tools in Python are weaker than in a strongly typed less dynamic language but will work more than well enough to give good heuristics for you here. Unless you use an unusually large pathological number of dynamic features (including adding and removing methods, intercepting method and property invocations, playing with import, manually modifying the namespace) - in which case any problems you have may well be associated with this dynamism...
Pylint picks up simpler problems, and will not detect problems with dynamic class/instance modifications and decorators - so it doesn't matter that the metric tools don't measure these
In any case, where you can usefully focus can be determined by much more than a dependency graph.
Heuristics for selecting code
I find that there are a number of different considerations for selecting code for improvement which work both individually and together. Remember that, at first, all you need to do is find a productive seam of work - you don't need to find the absolutely worst code before you start.
Use your judgement.
After a few cycles through the codebase, you will have a huge amount of information and be much better positioned to continue your work - if indeed more needs to be done.
That said, here are my suggestions:
High value to the business: For example any code that could cost your company a lot of money. Many of these may be obvious or widely known (because they are important), or they may be detected by running the important use cases on a system with the run-time profiler enabled. I use Coverage.
Static code metrics: There are a lot of metrics, but the ones that concern us are:
High afferent couplings. This is code that a lot of other files depends on. While I don't have a tool that directly outputs this, snakefood is a good way to dump the dependencies directly to file, one line per dependency, each being a tuple of afferent and efferent file. I hate to say it, but computing the afferent coupling value from this file is a simple exercise left to the reader.
High McCabe (cyclomatic) complexity: This is more complex code. PyMetrics seems to produce this measure although I have not used the tool.
Size: You can get a surprising amount of information by viewing the size of your project using a visualiser (eg https://superuser.com/questions/8248/how-can-i-visualize-the-file-system-usage-on-windows or https://superuser.com/questions/86194/good-program-to-visualize-file-system-usage-on-mac?lq=1. Linux has KDirStat at Filelight). Large files are a good place to start as fixing one file fixes many warnings.
Note that these tools are file-based. This is probably fine-enough resolution since you mention the project is itself has hundreds of modules (files).
Changes frequently: Code that changes frequently is highly suspect. The code may:
Historically have had many defects, and empirically may continue to do so
Be undergoing changes from feature development (high number of revisions in your VCS)
Find areas of change using a VCS visualisation tool such as those discussed later in this answer.
Uncovered code: Code not covered by tests.
If you run (or can run) your unit tests, your other automated tests and typical user tests with coverage, take a look at the packages and files with next to no coverage. There are two logical reasons why there is no coverage:
The code is needed (and important) but not tested at all (at least automatically). These areas are extremely high risk
The code may be unused and is a candidate for removal.
Ask other developers
You may be surprised at the 'smell' metrics you can gather by having a coffee with the longer-serving developers. I bet they will be very happy if someone cleans up a dirty area of the codebase where only the bravest souls will venture.
Visibility - detecting changes over time
I am assuming that your environment has a DVCS (such as Git or Mercurial) or at least a VCS (eg SVN). I hope that you are also using an issue or bug tracker of some kind. If so, there is a huge amount of information available. It's even better if developers have reliably checked in with comments and issue numbers. But how do you visualise it and use it?
While you can tackle the problem on a single desktop, it is probably a good idea to set up a Continuous Integration (CI) environment, perhaps using a tool like Jenkins. To keep the answer short, I will assume Jenkins from now on. Jenkins comes with a large number of plugins that really help with code analysis. I use:
py.test with JUnit test output picked up by the JUnit test report Jenkins plugin
Coverage with the Cobertura plugin
SLOCCount and SLOCCount plugin
Pylint and Violations plugin
Apparently there is a plugin for McCabe (cyclometric) complexity for Python, although I have not used it. It certainly looks interesting.
This gives me visibility of changes over time, and I can drill in from there. For example, suppose PyLint violations start increasing in a module - I have evidence of the increase, and I know the package or file in which this is occurring, so I can find out who's involved and go speak with them.
If you need historic data and you have just installed Jenkins, see if you can run a few manual builds that start at the beginning of the project and take a series of jumps forward in time until the present. You can choose milestone release tags (or dates) from the VCS.
Another important area, as mentioned above, is detecting the loci of changes in the code base. I have really liked Atlassian Fisheye for this. Besides being really good at searching for commit messages (eg bug id) or file contents at any point in time, it allows me to easily see metrics:
Linecount by directory and subdirectory
Committers at any point in time or in specific directories and/or files
Patterns of committal, both by time and also location in the source code
I'm afraid you are mostly on your own.
If you have decent set of tests, look at code coverage and dead code.
If you have a decent profiling setup, use that to get a glimpse of what's used more.
In the end, it seems you are more interested in fan-in/fan-out analysis, I'm not aware of any good tools for Python, primarily because static analysis is horribly unreliable against a dynamic language, and so far I didn't see any statistical analysis tools.
I reckon that this information is sort of available in JIT compilers -- whatever (function, argument types) is in cache (compiled) those are used the most. Whether or not you can get this data out of e.g. PyPy I really don't have a clue.
Source control tools can give a good indication of frequently updated modules - often indicating trouble spots.
If you don't have source control but the project is run from a shared location delete all the pycache folders or .pyc files. Over time/under use watch which files get recreated to indicate their use.
Analysing the Python imports printed when running from particular entry points with
python -v entry_point
may give some insight into which modules are being used. Although if you have known access points you should try the coverage module.
For a more intrusive solution, consider setting up project wide logging. You can log metrics easy enough, even over distributed programs.
I agree with the others, in that I have yet to come across a good runtime analysis tool for Python that will do this.
There are some ways to deal with it, but none are trivial.
The most robust, I think, would be to get the Python source and recompile the binaries with some sort of built-in runtime logging. That way you could just slide it into the existing environment without any code changes to your project.
Of course, that isn't exactly trivial to do, but it has the bonus that you might some day be able to get that merged back into the trunk for future generations and what not.
For non-recompile approaches, the first place I would look is the profile library's deterministic profiling section.
How you implement it will be heavily dependent on how your environment is set up. Do you have many individual scripts and projects run independently of one another, or just the one main script or module or package that gets used by everybody else, and you just want to know what parts of it can be trimmed out to make maintenance easier?
Is it a load once, run forever kind of set up, or a situation where you just run scripts atomically on some sort of schedule?
You could implement project-wide logging (as mentioned in #Hardbyte's answer), but that would require going through the project and adding the logging lines to all of your code. If you do that, you may as well just do it using the built-in profiler, I think.
Have a look at sys.setprofile: it allows you to install a profiler function.
Its usage is detailed in http://docs.python.org/library/profile.html#profile, for a jumpstart go here.
If you can not profile your application you will be bound to the cooverage approach.
Another thing you might have a look at is decorators, you can write a debugging decorator, and apply it to set of functions you suspect. Take alook here to see how to apply the decorator to an entire module.
You might also take a look at python call graph, while it will not generate quite what you want it shows you how often one function calls another:
If your code runs on user input this will be hard, since you would have to simulate 'typical' usage.
There is not more to tell you, just remember profiling as keyword.
Pylint sometimes gives warnings that (after careful consideration) are not justified. In which case it is useful to make use of the special #pylint: disable=X0123 comments (where X0123 is the actual error/warning message number) if the code cannot be refactored to not trigger the warning.
I'd like to second Hardbyte's mention of using your source control logs to see which files are most often changed.
If you are working on a system that has find, grep and sort installed, the following is a way to check which file imports what;
find . -name '*.py' -exec grep -EH '^import|^from .* import' {} \+| sort |less
To find the most popular imports across all files;
find . -name '*.py' -exec grep -Eh '^import|^from .* import' {} \+ | sort | less
These two commands should help you find the most-used modules from your project.

Debugging a scripting language like ruby

I am basically from the world of C language programming, now delving into the world of scripting languages like Ruby and Python.
I am wondering how to do debugging.
At present the steps I follow is,
I complete a large script,
Comment everything but the portion I
want to check
Execute the script
Though it works, I am not able to debug like how I would do in, say, a VC++ environment or something like that.
My question is, is there any better way of debugging?
Note: I guess it may be a repeated question, if so, please point me to the answer.
Your sequence seems entirely backwards to me. Here's how I do it:
I write a test for the functionality I want.
I start writing the script, executing bits and verifying test results.
I review what I'd done to document and publish.
Specifically, I execute before I complete. It's way too late by then.
There are debuggers, of course, but with good tests and good design, I've almost never needed one.
Here's a screencast on ruby debugging with ruby-debug.
Seems like the problem here is that your environment (Visual Studio) doesn't support these languages, not that these languages don't support debuggers in general.
Perl, Python, and Ruby all have fully-featured debuggers; you can find other IDEs that help you, too. For Ruby, there's RubyMine; for Perl, there's Komodo. And that's just off the top of my head.
There is a nice gentle introduction to the Python debugger here
If you're working with Python then you can find a list of debugging tools here to which I just want to add Eclipse with the Pydev extension, which makes working with breakpoints etc. also very simple.
My question is, is there any better way of debugging?"
Yes.
Your approach, "1. I complete a large script, 2. Comment everything but the portion I want to check, 3. Execute the script" is not really the best way to write any software in any language (sorry, but that's the truth.)
Do not write a large anything. Ever.
Do this.
Decompose your problem into classes of objects.
For each class, write the class by
2a. Outline the class, focus on the external interface, not the implementation details.
2b. Write tests to prove that interface works.
2c. Run the tests. They'll fail, since you only outlined the class.
2d. Fix the class until it passes the test.
2e. At some points, you'll realize your class designs aren't optimal. Refactor your design, assuring your tests still pass.
Now, write your final script. It should be short. All the classes have already been tested.
3a. Outline the script. Indeed, you can usually write the script.
3b. Write some test cases that prove the script works.
3c. Runt the tests. They may pass. You're done.
3d. If the tests don't pass, fix things until they do.
Write many small things. It works out much better in the long run that writing a large thing and commenting parts of it out.
Script languages have no differences compared with other languages in the sense that you still have to break your problems into manageable pieces -- that is, functions. So, instead of testing the whole script after finishing the whole script, I prefer to test those small functions before integrating them. TDD always helps.
There's a SO question on Ruby IDEs here - and searching for "ruby IDE" offers more.
I complete a large script
That's what caught my eye: "complete", to me, means "done", "finished", "released". Whether or not you write tests before writing the functions that pass them, or whether or not you write tests at all (and I recommend that you do) you should not be writing code that can't be run (which is a test in itself) until it's become large. Ruby and Python offer a multitude of ways to write small, individually-testable (or executable) pieces of code, so that you don't have to wait for (?) days before you can run the thing.
I'm building a (Ruby) database translation/transformation script at the moment - it's up to about 1000 lines and still not done. I seldom go more than 5 minutes without running it, or at least running the part on which I'm working. When it breaks (I'm not perfect, it breaks a lot ;-p) I know where the problem must be - in the code I wrote in the last 5 minutes. Progress is pretty fast.
I'm not asserting that IDEs/debuggers have no place: some problems don't surface until a large body of code is released: it can be really useful on occasion to drop the whole thing into a debugging environment to find out what is going on. When third-party libraries and frameworks are involved it can be extremely useful to debug into their code to locate problems (which are usually - but not always - related to faulty understanding of the library function).
You can debug your Python scripts using the included pdb module. If you want a visual debugger, you can download winpdb - don't be put off by that "win" prefix, winpdb is cross-platform.
The debugging method you described is perfect for a static language like C++, but given that the language is so different, the coding methods are similarly different. One of the big very important things in a dynamic language such as Python or Ruby is the interactive toplevel (what you get by typing, say python on the command line). This means that running a part of your program is very easy.
Even if you've written a large program before testing (which is a bad idea), it is hopefully separated into many functions. So, open up your interactive toplevel, do an import thing (for whatever thing happens to be) and then you can easily start testing your functions one by one, just calling them on the toplevel.
Of course, for a more mature project, you probably want to write out an actual test suite, and most languages have a method to do that (in Python, this is doctest and nose, don't know about other languages). At first, though, when you're writing something not particularly formal, just remember a few simple rules of debugging dynamic languages:
Start small. Don't write large programs and test them. Test each function as you write it, at least cursorily.
Use the toplevel. Running small pieces of code in a language like Python is extremely lightweight: fire up the toplevel and run it. Compare with writing a complete program and the compile-running it in, say, C++. Use that fact that you can quickly change the correctness of any function.
Debuggers are handy. But often, so are print statements. If you're only running a single function, debugging with print statements isn't that inconvenient, and also frees you from dragging along an IDE.
There's a lot of good advice here, i recommend going through some best practices:
http://github.com/edgecase/ruby_koans
http://blog.rubybestpractices.com/
http://on-ruby.blogspot.com/2009/01/ruby-best-practices-mini-interview-2.html
(and read Greg Brown's book, it's superb)
You talk about large scripts. A lot of my workflow is working out logic in irb or the python shell, then capturing them into a cascade of small, single-task focused methods, with appropriate tests (not 100% coverage, more focus on edge and corner cases).
http://binstock.blogspot.com/2008/04/perfecting-oos-small-classes-and-short.html

Categories

Resources