As a personal project (in order to learn python better), I starter working on a duplicate file remover (especially for .mp3 files since I thought of it while trying to organise my full-of-duplicates music collection). Now, I'm fairly clear on how to proceed, matching file names and offering for deletion only those that present more that 0.7 similarity ratio, and using md5 sums for those files that are the same but have completely different names (eg: "metallica-nothing else matters" and "Track1"). The problem is that I don't know what to do about those files that have different names and they are a bit different from one another, for example, "nothing else matters" and "Track1" are the same except for the fact that "Track1" has 2 seconds of silence at the end. My question is: Is there some kind of method or algorithm that checks similarities between files themselves? Something like string matching but on files? Doesn't matter if it's a complicated algorithm, the harder the better since I'm doing this only to learn :D
You could use Chromaprint, that computes a fingerprint for a piece of music. It should be able to find similar music files.
If you want to push this further, you could use the api of musicbrainz to find the exact information about a piece of music.
These libraries are used in two greats music library tagging and sorting applications I use : picard and beets.
you can also look at win32 module, here is the link
http://timgolden.me.uk/python/index.html
Related
I have been studying programming for a few years and I am now working on my first desktop application. I am making a simple program that is able to keep track of information pertaining to a DND (Dungeons and Dragons) character/s. I want to find a way to store information about these characters so they next time the applications is launched, the characters will be saved. How do things like spotify save information about each user? First, I will give some info about the program itself. I have written it in python and it is organized as follows:
I have a file, which serves as the brain of the application (app.py).
A file which defines a class representing a character
A file defining a class that is used to find information about the characters
Other files defining classes used the build the UI
So far in my studies, I have only gathered inputted information from txt files, input functions and APIs via requests. I have worked with JSON before and am thinking this may be an option, but I am not sure how this would work in this case. I also had the idea of storing data in txt files, but want to learn the way it is done in the real world in order to make the best use of my time.
TLDR: I am making a desktop application using python and want an effective and common way people store information they want to access the next time the program is ran. I am looking for a local way to save the data that is also SAFE. If you have a recommendation that is server/cloud based, I would still like to hear how it may be done that as I am sure the knowledge will still be beneficial. I am looking for a way to SAFELY store information that will be saved even after the application is terminated. Any advice or anything you have personally used is appreciated.
I have a set of files (compiled software) that I want to give an unique fingerprint before distribution. The idea is to write a script that:
Randomly generates a character sequence
Appends the character sequence to a file in the project
Stores the fingerprint in a database with the addressee
Distributes the software to the addressee
The requirements for the fingerprint process is that:
The fingerprint is difficult to detect (i.e. not stored in the file metadata or easily accessible areas)
The fingerprint does not corrupt the data of the file the sequence is added to
The fingerprint can be added to an executable or dll file
It's easy to read the fingerprint if you know where to look
Are there any open source solutions that is built for the purpose of fingerprinting files?
Storing information in the file without corrupting it and in a way that is not easily detectable is an exercise in steganography, and quite a hard one. This theoretical tool needs to be able to parse executable structure, and properly modify it, edit offsets if needed, or detect padding arias, or basically do some of the work that the compiler is doing. I doubt that it exists or is reliable.
However, there are quite a few steganography tools that can store information in pictures by subtly changing the colors of the pixels, perhaps you can store your information in the icon of the exe file or any included asset.
Another way is to hide the data at compilation time, in optimization level of the performance-uncritical parts of the executable, so that compiler generates slightly different code, but the behavior is guaranteed to stay consistent. You can now use file hashes as your fingerprint.
Yet another way is to just create unused string inside some random function, mark it as volatile or analog in your language of choice to prevent the compiler from optimizing it out of your program and put something noticeable in it, like REPLACE_ME. Now you can open this file, search for this string and replace it with the identifier that you have generated. If identifier and the string were the same length - you can’t damage your software.
Another, more subtle way is to create multiple different rephrasings of the same messages in your app and swap them in and out as a way to differentiate versions. If your programming language stores null-terminated strings then this is very easy, just make your strings in the code as long as the longest rephrasing. If your language stores length of the string then you have to dynamically recalculate it too.
Alternatively, if you are working with the Unicode strings in your code, then you can use similar-looking glyphs in some strings as a less effort version of previous idea. Basically you are performing a homograph attack on your strings. Alternatively you can use unicode control chars (ZWJ, ZWNJ, etc.) that do not affect most languages and are invisible.
All schemes is easily discovered by diffing two different distributions of the software, the one with the different optimization levels could be plausibly written off as just different builds of the software, but the persistent attacker still could figure it out.
Since you are talking about compiled software, maybe an alternative solution could be to use an execbinary encrypting tool. When you execute the file it will ask for a password, if it's correct then it will use the password to generate a key. Then it uses that key to decrypt the program directly in memory. That way they won't be able to analyze the binary and even with the key it would be a lot more difficult to do so, much less modify it. You can put as many fingerprints as you like, regular text strings, into the code and they will most likely stay there.
I have two .csv files that I want to use LibreOffice's compare documents tool (Edit>Compare Document) with.
These csv files are made after the run of a long and involved script, and it would be nice to be able to have the compare process to be automatic as well, with the result being a window of LibreOffice open with the changes as if I selected compare manually. I want the specific LibreOffice gui (which I believe does a great job highlighting differences) not just a diff.
Looking online, it seems like there is nice but limited set of python wrappers for libre office (pyoo).
However, despite related questions, I couldn't see any way of gaining access to the compare functionality through this or any other library. Is the Compare Documents functionality available at the python level, the UNO API level, or simply not available at all?
Use the dispatcher:
Dispatcher.executeDispatch(
(XDispatchProvider)Frame, ".uno:CompareDocuments", "", 0, propertyValueFile);
A complete Java example is at https://forum.openoffice.org/en/forum/viewtopic.php?f=44&t=2795.
I'd like to seperate the channels of a mp3 file in Python and save it in two other files.
Does anybody know a library for this.
Thanks in advance.
I assume you want to split the channels losslessly, without decoding MP3 and re-encoding it - otherwise you would not have mentioned MP3 at all and would have easily found many tools like Audacity to do that.
There are 4 channel modes of MP3 frames - this means 4 types of MP3 files: simple stereo, joint-stereo, dual-channel, mono. joint-stereo files can't be split without loss. mono files doesn't need splitting. The rest: stereo and dual-channel, consists of less than 0.1% of all MP3 files, technically can be split into 2 files, each for a channel, without loss. However there is not any tool on the Internet to do that - not any command line tool nor any GUI tool, because few need the function.
There are not any python library for you neither. Most libraries abstracted MP3 files into a common audio which you can manipulate, after decoding. pymad is the only one specific to MP3 file, and it can tell if a file is using any of the 4 channel modes, but does not offer to extract a channel without decoding it. If you write a new tool, you will have to work on raw MP3 files or produce a library for it.
And it is not easy to write a tool or library for it. It's one stream with 2 channels and not two streams interleaved on a frame level. You cannot simply work on MP3 frames, drop some frames, keep others, and manage to extract a channel out that way. It's a task for a professional, and perhaps best happen in a decoder project (like lame or libmad) and not in a file manipulation project (like mp3info or the python eyeD3). In other words, this feature is likely written in C, not python.
Implementaiton Note:
The task to build such a tool thus suits well for a computer science C-programming language course project:
1. it takes a lot of time to do;
2. it requires every skill learned from C programming course;
3. it can get wrong easily;
4. it is likely built on the work of other projects, a lesson of adaptating existing work;
5. is a damn-hard endeavor that no-one did before and thus very rewarding
6. perhaps can be done in 300 difficult lines of code instead of bloated simple Visual Basic code, thus is a good lession of modesty and quality;
7. and finally: nobody is waiting in an hurry for a working implementation.
All condition fits perfectly for a C-programming course project.
Implementation Note 2:
some bit-rates are only possible in mono mode (80kbps), and some bit-rates are only possible in stereo mode (e.g. 320kpbs). Luckily this does not present a problem in this task, because all dual-mp3 bit-rate can be mapped into a fitting mono-mp3 bit-rate -- but not vice versa!
I am looking for some general advice about the mp3 format before I start a small project to make sure I am not on a wild-goose chase.
My understanding of the internals of the mp3 format is minimal. Ideally, I am looking for a library that would abstract those details away. I would prefer to use Python (but could be convinced otherwise).
I would like to modify a set of mp3 files in a fairly simple way. I am not so much interested in the ID3 tags but in the audio itself. I want to be able to delete sections (e.g. drop 10 seconds from the 3rd minute), and insert sections (e.g. add credits to the end.)
My understanding is that the mp3 format is lossy, and so decoding it to (for example) PCM format, making the modifications, and then encoding it again to MP3 will lower the audio quality. (I would love to hear that I am wrong.)
I conjecture that if I stay in mp3 format, there will be some sort of minimum frame or packet-size to deal with, so the granularity of the operations may be coarser. I can live with that, as long as I get an accuracy of within a couple of seconds.
I have looked at PyMedia, but it requires me to migrate to PCM to process the data. Similarly, LAME wants to help me encode, but not access the data in place. I have seen several other libraries that only deal with the ID3 tags.
Can anyone recommend a Python MP3 library? Alternatively, can you disabuse me of my assumption that going to PCM and back is bad and avoidable?
If you want to do things low-level, use pymad. It turns MP3s into a buffer of sample data.
If you want something a little higher-level, use the Echo Nest Remix API (disclosure: I wrote part of it for my dayjob). It includes a few examples. If you look at the cowbell example (i.e., MoreCowbell.dj), you'll see a fork of pymad that gives you a NumPy array instead of a buffer. That datatype makes it easier to slice out sections and do math on them.
I got three quality answers, and I thank you all for them. I haven't chosen any as the accepted answer, because each addressed one aspect, so I wanted to write a summary.
Do you need to work in MP3?
Transcoding to PCM and back to MP3 is unlikely to result in a drop in quality.
Don't optimise audio-quality prematurely; test it with a simple prototype and listen to it.
Working in MP3
Wikipedia has a summary of the MP3 File Format.
MP3 frames are short (1152 samples, or just a few milliseconds) allowing for moderate precision at that level.
However, Wikipedia warns that "Frames are not independent items ("byte reservoir") and therefore cannot be extracted on arbitrary frame boundaries."
Existing libraries are unlikely to be of assistance, if I really want to avoid decoding.
Working in PCM
There are several libraries at this level:
LAME (latest release: October 2017)
PyMedia (latest release: February 2006)
PyMad (Linux only? Decoder only? Latest release: January 2007)
Working at a higher level
Echo Nest Remix API (Mac or Linux only, at the moment) is an API to a web-service that supports quite sophisticated operations (e.g. finding the locations of music beats and tempo, etc.)
mp3DirectCut (Windows only) is a GUI that apparently performs the operations I want, but as an app. It is not open-source. (I tried to run it, got an Access Denied installer error, and didn't follow up. A GUI isn't suitably for me, as I want to repeatedly run these operations on a changing library of files.)
My plan is now to start out in PyMedia, using PCM.
Mp3 is lossy, but it is lossy in a very specific way. The algorithms used as designed to discard certain parts of the audio which your ears are unable to hear (or are very difficult to hear). Re-doing the compression process at the same level of compression over and over is likely to yield nearly identical results for a given piece of audio. However, some additional losses may slowly accumulate. If you're going to be modifying files a lot, this might be a bad idea. It would also be a bad idea if you were concerned about quality, but then using MP3 if you are concerned about quality is a bad idea over all.
You could construct a test using an encoder and a decoder to re-encode a few different mp3 files a few times and watch how they change, this could help you determine the rate of deterioration and figure out if it is acceptable to you. Sounds like you have libraries you could use to run this simple test already.
MP3 files are composed of "frames" of audio and so it should be possible, with some effort, to remove entire frames with minimal processing (remove the frame, update some minor details in the file header). I believe frames are pretty short (a few milliseconds each) which would give the precision you're looking for. So doing some reading on the MP3 File Format should give you enough information to code your own python library to do this. This is a fair bit different than traditional "audio processing" (since you don't care about precision) and so you're unlikely to find an existing library that does this. Most, as you've found, will decompress the audio first so you can have complete fine-grained control.
Not a direct answer to your needs, but check the mp3DirectCut software that does what you want (as a GUI app). I think that the source code is available, so even if you don't find a library, you could build one of your own, or build a python extension using code from mp3DirectCut.
As for removing or extracting mp3 segments from an mp3 file while staying in the MP3 domain (that is, without conversion to PCM format and back), there is also the open source package PyMp3Cut.
As for splicing MP3 files together (adding e.g. 'Credits' to the end or beginning of an mp3 file) I've found you can simply concatenate the MP3 files providing that the files have the same sampling rate (e.g. 44.1khz) and the same number of channels (e.g. both are stereo or both are mono).