Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
Is new macbook m1 suitable for Data Science?
Do Data Science python libraries such as pandas, numpy, sklearn etc work on the macbook m1 (Apple Silicon) chip and how fast compared to the previous generation intel based macbooks?
This GitHub repository has lots of useful information about the Apple M1 chip and data science in Python https://github.com/neurolabusc/AppleSiliconForNeuroimaging. I have included representative quotes below. This repository focuses on software for brain imaging analysis, but the takeaways are broad.
Updated on 27 September 2021.
TL;DR
Unless you are a developer, I would strongly discourage scientists from purchasing an Apple Silicon computer in the short term. Productive work will require core tools to be ported. In the longer term, this architecture could have a profound impact on science. In particular if Apple develops servers that exploit the remarkable power efficiency of their CPUs (competing with AWS Graviton) and leverage the Metal language and GPUs for compute tasks (competing with NVidia's Tesla products and CUDA language).
Limitations facing Apple Silicon
The infrastructure scientists depend on is not yet available for this architecture. Here are some of the short term limitations:
Native R can use the unstable R-devel 4.1. However, RStudio will require gdb.
Julia does not yet natively support Apple Silicon.
Python natively supports Apple Silicon. However, some modules have issues or are slow. See the NiBabel section below.
Scientific modules of Python, R, and Julia require a Fortran compiler, which is currently only available in experimental form.
While Apple's C Clang compiler generates fast native code, many scientific tools will need to wait until gcc and gFortran compilers are available.
Tools like VirtualBox, VMware Fusion, Boot Camp and Parallels do not yet support Apple Silicon. Many users rely on these tools for using Windows and Linux programs on their macOS computers.
Docker can support Apple Silicon. However, attempts to run Intel-based containers on Apple Silicon machines can crash as QEMU sometimes fails to run the container. These containers are popular with many neuroimaging tools.
Homebrew 3.0 supports Apple Silicon. However, many homebrew components do not support Apple Silicon.
MATLAB is used by many scientific tools, including SPM. While Matlab works in translation, it is not yet available natively (and mex files will need to be recompiled).
FSL and AFNI do not yet natively support this architecture. While code may work in translation, creating some native tools must wait for compilers and libraries to be updated. This will likely require months.
The current generation M1 only has four high performance cores. Most neuroimaging pipelines combine sequential tasks that only require a single core (where the M1 excels) as well as parallel tasks. Those parallel tasks could exploit a CPU with more cores (as shown in the pigz and niimath tests below). Bear in mind that this mixture of serial and parallel code faces Amdahls law, with diminishing returns for extra cores.
The current generation M1 has a maximum of 16 Gb of RAM. Neuroimaging datasets often have large memory demands (especially multi-band accelerated functional, resting-state and diffusion datasets).
In general, the M1 and Intel-based Macs have identical OpenGL compatibility, with the M1 providing better performance than previous integrated solutions. However, there are corner cases that may break OpenGL tools. Here I describe four limitations. First, OpenGL geometry shaders are not supported (there is no Metal equivalent). Second, the new retina displays support wide color with 16 bitsPerSample that can cause issues for code that assumes 32-bit RGBA textures (such as the text in this Apple example code). Third, textures can be handled differently. Fourth, use of the GL_INT_2_10_10_10_REV data type will cripple performance (tested on macOS 11.2). This is unfortunate, as Apple advocated for this datatype once upon a time. In this case, code msut be changed to use the less compact GL_HALF_FLOAT which is natively supported by the M1 GPU. This impacts neuroimaging scientists visualizing DTI tractography where GPU resources can be overwhelmed.
Related
I'm currently working on building a graphics library with Python. (I know opengl exists, but this is just for fun) Essentially, I want to be able to implement many of the common features available with opengl.
Right now, I'm using pyopencl which uses opencl under the hood to send code to the GPU for processing.
Here's the kicker though, I'm developing on a 2015 MacBook Pro that has an AMD GPU.
I've written all of the code to make this work and now I'm working on finding the bottlenecks in the code to try to optimize.
I've found that the biggest bottle neck is in my fragment shader implemenetation when I'm converting numpy arrays to cl.Buffer objects prior to GPU processing and then back to numpy arrays after processing.
Doing some research has led me to think that using SVMs would help to minimize this cost, but.... SVMs were introduced in opencl 2.0. Apparently, Apple has stopped supporting opencl in favor of their own in house GPU library Metal and so I'm stuck with opencl 1.2
So I guess my question is, has anyone else hit this roadblock and if so, what is the common way of handling it? If I transition to Metal, then my code is no longer as universal, but if I stay on opencl, I have performance problems. I could autodetect the platform that the code is being run on and use different implementations specific to the platform, but one of my problems is I don't know if there is a trusted standard implementation for Metal in Python.
Can I somehow use my AMD processor to speed up computations in my Python script? I'm doing object detection using OpenCV 4.0 with cv2.dnn module.
Basing on similar questions I've tried to use cv2.UMat but it doesn't speed up computations, so I assume that the script was still run on my poor CPU.
GPU info: Advanced Micro Devices, Inc. [AMD/ATI] Thames [Radeon HD 7500M/7600M Series]
Sorry but AFAICT, that's a fairly old GPU (pre-GCN architecture). Those were not really suited for GPGPU computations. It should be still possible to install OpenCL drivers, but i cannot guarantee anything as i can't try it.
I assume you're on linux. Here is an article on how to use older AMD GPUs with Linux. You would have to follow steps in "Running Open CL on old cards" section, which involve installing Ubuntu 14.04 and fglrx 15.12 drivers.
I do a lot of coding in Python (Anaconda install v. 3.6). I don't compile anything, I just run machine learning models (mainly sci-kit and tensor flow) Are there any issues with running these on an workstation with AMD chipset? I've only used Intel before and want to make sure I don't buy wrong.
If it matters it is the AMD Ryzen 7-1700 processor.
Are you asking about compatibility or performance?
Both AMD and Intel market CPU products compatible with x86(_64) architecture and are functionally compatible with all software written for it. That is, they will run it with high probability (there always may be issues when changing hardware, even while staying with the same vendor, as there are too many variables to account).
Both Intel and AMD offer a huge number of products with widely varying level of marketed performance. Performance of any application is determined not only by a chosen vendor of a central processor, but by a huge number of other factors, such as amount and speed of memory, disk, and not the least the architecture of the application itself. In the end, it is only real-world measurements that decide, but some estimations can be made by looking at relevant benchmarks and understanding underlying principles of computer performance.
Since the Yocto Linux distribution can run on both machines, I'm assuming it would have no trouble compiling and using any language, which ordinary developers with a Linux system would use. Am I right in making this assumption?
It says, on the Intel page, that compatible languages are:
C/C++, Python, Node.js, HTML5, JavaScript
Shouldn't these languages be compatible on a Linux system? Just install the compiler on Linux and you should be fine, no?
The only explanation that comes to mind is that these languages have libraries specifically written to interact with Arduino hardware.
If this is the case, which languages are strongest in terms of resources, libraries, compatibility, etc.?
Also, please, correct me if I said anything marginally wrong. Thanks for any help, hugely appreciated.
I believe you are referring to the documentation for IoT Developer Kit. The IoT devkit is solution comprised of various hardware and software options to create IoT projects using Intel's maker boards such as Intel Edison and Intel Galileo. It includes a set of I/O and Sensor libraries specifically libmraa and upm currently available for C/C++, Python and JavaScript.
Libmraa provides APIs to interface with the I/O on board. With board detection done at runtime, you can create portable code that works across multiple platforms.
UPM is more of a high-level repository of sensors that uses mraa. You can find code samples for various sensors currently supported which helps in speeding up the development time.
Recently Java is also added to the list of supported languages, you can find samples in the repository.
What is the problem in developing some languages, for example python for some optimized techniques with some of LLVM / Parrot.
PyPy, LLVM, Parrot are the main technologies for common platform development.
I see this like:
PyPy - framework to build VM with build in optimized VM for python
So it quite general solution. The process goes as listed down:
dynamic_language_code ->
PyPy frontend ->
PyPy internal code - bytecode ->
PyPy optimization ->
leaving PyPy code and:
a. PyPy backend for some VM (like jvm)
b. som Kit to make own VM
c. processing/running PyPy internal code
Am I right About this process? For python there is optimized VM? Particularly by default there is build in VM for optimized PyPy code (step 5.c) - which is for python and every language processing can stop there and be running by it?
Parrot - much like PyPy, but without 5.a and 5.b ? Some internal improvements for dynamic processing (Parrot Magic Cookies).
Both Parrot and PyPy are designed to create a platform which create a common dynamic languages runtime, but PyPy wants more - also to create more VM.
Where is the sens of PyPy? For what we need to create more VM? Shouldn't be better to focus on one VM (like in parrot) - because there is common one code level - either PyPy internal bytecode or Parrot ones.
I think we can't gain nothing better to translate to PyPy bytecode to newly created with PyPy VMs.
LLVM - i see this very similar to PyPy but without VM generator.
It is mature, well designed environment with similar targets as PyPy (but without VM generator) but working on low level structure and great optimization/JIT techniques implemeted
Is see this as: LLVM is general use, but Parrot and **PyPy* designed for dynamic languages. In PyPy / Parrot is more easy to introduce some complicated techniques - because it is more high level then LLVM - like sophisticate compiler which can better understand high level code and produce better assembler code (which humans can't write in reasonable time), then the LLVM one?
Questions:
Am I right? Is there any reason that porting some dynamic language would be better to llvm then to for example Parrot?
I haven't see the activity on development python on Parrot. Is it because using python C extensions doesn't work on parrot? The same problem is in PyPy
Why other VMs don't want to move to LLVM / parrot. Eg ruby -> parrot, CLR/ JVM -> LLVM. Wouldn't be better for them to move to more sophisticated solution? LLVM is in high development process and has big companies investing in.
I know the problem might be in recompile are resources, if there is need to change bytecode - but it is not obligatory - as we can try to port old bytecode to new one, and new compilers produce new bytecode (never less java still need to interpreted own bytecode - so the frontend can check it and translate it to new bytecode)?
What are the problems with linking for example jvm libraries inside llvm (if we port somehow java/jvm/scala to llvm)?
Can you correct me if i'm wrong somewhere
Some addings:
How does Parrot compare to other virtual machines
What's the benefit of Parrot VM for end-users
What are the differences between LLVM and java/jvm
=============
CLARIFICATION
I want to figure how all this software consist - and what is the problem to porting one to other.
That not stuff anybody can possible answer in a stackoverflow questions but i give it a minmal shot.
First what problems do the 3 projects solve?
pypy allows you to implement an interpreter in a high level language and you get a generated jit for free. The good thing about this is that you don't have a dependence mismatch between the langauge and the platform. Thats the reason why pypy-clr is faster then IronPython.
More info here: http://codespeak.net/pypy/dist/pypy/doc/extradoc.html --> High performance implementation of Python for CLI/.NET with JIT compiler generation for dynamic)
llvm is a low level infrastructure for compilers. The general idea is to have one "high level assembly". All the optomizations work on that language. Then there is tons of infrastructure around to help you build compilers (JIT or AOT). Implementing a dynamic language on llvm is possible but needs more work then implementing it on pypy or parrot. You, for example, can't get a GC for free (there are GC you can use together with LLVM see http://llvm.org/devmtg/2009-10/ --> the vmkit video ) There are attempts to build a platform better for dynamic langauges based on llvm: http://www.ffconsultancy.com/ocaml/hlvm/
I don't know that much about parrot but as I understand they want to build one standard VM specialized for dynamic langauges (perl, php, python ....). The problem here is the same as with compiling to JVM/CLR there is a dependency missmatch, just a much smaller one. The VM still does not know the semantics of your langauge. As I unterstand parrot is still pretty slow for user code. (http://confreaks.net/videos/118-elcamp2010-parrot)
The answer to your question:
Am I right? Is there any reason that porting some dynamic language would be better to llvm then to for example Parrot?
Thats a question of effort. Building everthing your self and specialized for you will eventually be faster but it's a LOT more effort.
I haven't see the activity on development python on Parrot. Is it because using python C extensions doesn't work on parrot? The same problem is in PyPy.
Targeting parrot would (at this point) not likely have a benefit over pypy. Why nobody else does it I don't know.
Why other VMs don't want to move to LLVM / parrot. Eg ruby -> parrot, CLR/ JVM -> LLVM. Wouldn't be better for them to move to more sophisticated solution? LLVM is in high
development process and has big companies investing in.
Ok there is a lot of stuff in that question.
Like I said LLVM is hard to move to and parrot is not that fast (correct me if im wrong).
Ruby has Rubinius witch tries to do a lot in ruby and jits to llvm (http://llvm.org/devmtg/2009-10/ --> Accelerating Ruby with LLVM).
There is a implementation of CLR/JVM on LLVM but they both already have very mature implemantations that have big investments.
LLVM is not high level.
I know the problem might be in recompile are resources, if there is need to change bytecode - but it is not obligatory - as we can try to port old bytecode to new one, and new compilers produce new bytecode (never less java still need to interpreted own bytecode - so the frontend can check it and translate it to new bytecode)?
I have no idea what the question is.
What are the problems with linking for example jvm libraries inside llvm (if we port somehow java/jvm/scala to llvm)?
Watch the video of VMKit I linked above that show how far they got and what the problem is (and how they solved it).
Can you correct me if i'm wrong somewhere
Lots of stuff you wrote is wrong or I just don't anderstand what you mean, but the stuff I linked should make a lot of stuff clearer.
Some examples:
Clojure
The creater didn't want all the work of implementing his own vm and all the libraries. So where to go to? Since Clojure is a new langauge you can build it in a way that works well on a platform like the JVM by restricting a lot of dynamic stuff a language like python or ruby would have.
Python
The language can't (practically) be changed to work well on JVM/CLR. So implementing python on those wont bring massive speedups. Static compiler won't work very well either because there are not many static guarantees. Writing a JIT in C will be fast but very hard to change (see the psyco project). Using the llvm jit could work and is explored by the Unladen Swallow project (again http://llvm.org/devmtg/2009-10/ --> Unladen Swallow: Python on LLVM). Some people wanted to have python in python so they started pypy and their idea seams to work really well (see above). Parrot could work as well but I have not seen anybody have try (feel free).
On everything:
I think you're confused and I can understand that. Take your time and read, listen, watch everything you can get. Don't stress yourself. There are a lot of parts to this and eventually you see how what fits together and what makes sense and even when you know a lot there is still a lot of discussing one may do. The question is where to implement a new language or how to speed up a old language have many answers and if you ask 3 people you're likely to get three different answers.
What are you trying to implement? Your question is very confusingly worded (I realize English is likely not your first language).
LLVM and PyPy are both mature, useful projects, but really don't overlap much at this point. (At one point, PyPy could generate LLVM bytecode—which was statically compiled to an interpreter—as opposed to C code, but it didn't provide much of a performance benefit and is no longer supported.)
PyPy lets you write an interpreter in RPython and use that as a description to generate a native code interpreter or JIT; LLVM is a C++ framework for building a compiler toolchain which can also be used to implement a JIT. LLVM's optimizers, code generation and platform support are significantly more advanced than those of PyPy, but it isn't as well suited to building a dynamic language runtime (see the Unladen Swallow retrospective for some examples of why). In particular, it is not as effective at collecting/using runtime feedback (which is absolutely essential for making dynamic languages perform well) as PyPy's trace-based JIT. Also, LLVM's garbage collection support is still somewhat primitive, and it lacks PyPy's unique ability to automatically generate a JIT.
Incidentally two Java implementations are built on LLVM—J3/VMKit and Shark.
You might consider watching the PyPy talk from Stanford last week; it provides a pretty decent overview of how PyPy works. Carl Friedrich Bolz's presentation also provides a good overview of the state of VM implementation.
The main reason? Because VM design is not a settled technology, and having a variety of VMs with different goals and objectives allows a variety of mechnisms to be tried in parallel rather than all having to be tried in series.
The JVM, CLR, PyPy, Parrot, LLVM and the rest all target different kinds of problems in different ways. It's similar to the reasons why Chrome, Firefox, Safari and IE all use their own Javascript engines.
Unladen Swallow attempted to apply LLVM to CPython, and spent more of their time fixing issues in LLVM than they did in doing anything Python specific.
Python-on-Parrot suffered from semantic differences between Perl 6 and Python causing problems with the front-end compilation process, so future efforts in this area are likely to use the PyPy front-end to target the Parrot VM.
Different VM developers certainly keep an eye on what the others are doing, but even when they lift good ideas they will put their own spin on them before incorporating them.