Develop Reference

How to run the Atmospheric Correction Model in Google Earth Engine

I'm trying to run the atmospheric correction for Landsat images in Google Earth Engine. In the region I'm interested loads of images are only available at TOA reflection and I'd like to approximate the surface reflection as good as possible.
I'm considering to use Sam Murphy's set up that uses Py6S available on github. I got stuck running the docker container.
I went through the following steps:
Enable Virtual Machine
Enable Microsoft Hyper V in powershell
Install Docker - Docker login - test docker: open cmd as admin - run: docker run hello-world
The Docker file (downloaded from github) is located in a folder called D:\ProjectFolder\GEE\preprocess\ee-atmcorr-timeseries-master. So after opening cmd as admin and trying to run the following code:
docker build
D:\ProjectFolder\GEE\preprocess\ee-atmcorr-timeseries-master -t
atmcorr-timeseries
I get an error (even after allowing Deamon to run in experimental mode):
Error:
The command '/bin/sh -c conda install -c anaconda pip && pip install earthengine-api openpyxl oauth2client' returned a non-zero code: 1
Below is the lengthy error report. I'm on a Windows 10 machine (version 1803) 64x. I had Anaconda and Python previously installed but figured this shouldn't affect the result.
I'm new to docker and the interaction with google earth engine so sorry for lengthy (but perhaps still incomplete) question.
I figured It had to do with incorrect calling of the locations on my drives (Anaconda/Python in C while the Model is on a D drive) but i'm not sure and haven't been able to fix it by changing the working directory (cd command in cmd).
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/miniconda/lib/python3.6/site-packages/conda/exceptions.py", line 819, in __call__
return func(*args, **kwargs)
File "/miniconda/lib/python3.6/site-packages/conda/cli/main.py", line 78, in _main
exit_code = do_call(args, p)
File "/miniconda/lib/python3.6/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
exit_code = getattr(module, func_name)(args, parser)
File "/miniconda/lib/python3.6/site-packages/conda/cli/main_install.py", line 11, in execute
install(args, parser, 'install')
File "/miniconda/lib/python3.6/site-packages/conda/cli/install.py", line 235, in install
force_reinstall=context.force,
File "/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 518, in solve_for_transaction
force_remove, force_reinstall)
File "/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 451, in solve_for_diff
final_precs = self.solve_final_state(deps_modifier, prune, ignore_pinned, force_remove)
File "/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 180, in solve_final_state
index, r = self._prepare(prepared_specs)
File "/miniconda/lib/python3.6/site-packages/conda/core/solve.py", line 592, in _prepare
self.subdirs, prepared_specs)
File "/miniconda/lib/python3.6/site-packages/conda/core/index.py", line 215, in get_reduced_index
new_records = query_all(spec)
File "/miniconda/lib/python3.6/site-packages/conda/core/index.py", line 184, in query_all
return tuple(concat(future.result() for future in as_completed(futures)))
File "/miniconda/lib/python3.6/site-packages/conda/core/subdir_data.py", line 95, in query
self.load()
File "/miniconda/lib/python3.6/site-packages/conda/core/subdir_data.py", line 149, in load
_internal_state = self._load()
File "/miniconda/lib/python3.6/site-packages/conda/core/subdir_data.py", line 246, in _load
_internal_state = self._process_raw_repodata_str(raw_repodata_str)
File "/miniconda/lib/python3.6/site-packages/conda/core/subdir_data.py", line 369, in _process_raw_repodata_str
info['fn'] = fn
TypeError: 'NoneType' object does not support item assignment
`$ /miniconda/bin/conda install -c anaconda pip`
environment variables:
CIO_TEST=<not set>
CONDA_ROOT=/miniconda
PATH=/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin
:/bin
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
active environment : None
user config file : /root/.condarc
populated config files : /root/.condarc
conda version : 4.5.11
conda-build version : not installed
python version : 3.6.6.final.0
base environment : /miniconda (writable)
channel URLs : https://conda.anaconda.org/anaconda/linux-64
https://conda.anaconda.org/anaconda/noarch
https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : /miniconda/pkgs
/root/.conda/pkgs
envs directories : /miniconda/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/4.5.11 requests/2.20.0 CPython/3.6.6 Linux/4.14.23-linuxkit ubuntu/16.04 glibc/2.23
UID:GID : 0:0
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.

cookiecutter data science templet

I installed cookiecutter using
`conda install cookiecutter`
but when I try to import the repo from git hub using
`cookiecutter https://github.com/drivendata/cookiecutter-data-science`
I got the following error message:
{
Traceback (most recent call last):
File "C:\Users\iamuraptha\Anaconda3\Scripts\cookiecutter-script.py", line 5, in <module>
sys.exit(cookiecutter.cli.main())
File "C:\Users\iamuraptha\Anaconda3\lib\site-packages\click\core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "C:\Users\iamuraptha\Anaconda3\lib\site-packages\click\core.py", line 697, in main
rv = self.invoke(ctx)
File "C:\Users\iamuraptha\Anaconda3\lib\site-packages\click\core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\iamuraptha\Anaconda3\lib\site-packages\click\core.py", line 535, in invoke
return callback(*args, **kwargs)
File "C:\Users\iamuraptha\Anaconda3\lib\site-packages\cookiecutter\cli.py", line 120, in main
password=os.environ.get('COOKIECUTTER_REPO_PASSWORD')
File "C:\Users\iamuraptha\Anaconda3\lib\site-packages\cookiecutter\main.py", line 63, in cookiecutter
password=password
File "C:\Users\iamuraptha\Anaconda3\lib\site-packages\cookiecutter\repository.py", line 103, in determine_repo_dir
no_input=no_input,
File "C:\Users\iamuraptha\Anaconda3\lib\site-packages\cookiecutter\vcs.py", line 99, in clone
stderr=subprocess.STDOUT,
File "C:\Users\iamuraptha\Anaconda3\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "C:\Users\iamuraptha\Anaconda3\lib\subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'clone', 'https://github.com/drivendata/cookiecutter-data-science']' returned non-zero exit status 128.
}

You need something git can clone. Use https://github.com/drivendata/cookiecutter-data-science.git (this is the link you get when you click the 'clone or download' button in Github)

I just wanted to add a clarification for people coming here because they have a somewhat similar problem. The problem is not that the address isn't a valid git link. You can give cookiecutter a regular URL to a github repo and it will work, provided the directory in which cookiecooker is installed is in your environment's path. I'm not on Windows so I can't confirm, but check the documentation for more details about adjusting the path. And remember to factor Anaconda in if you are using it instead of the system Python.
For me this was a straightforward process on a Mac because I followed the prompt when I installed Anaconda, but I needed to add the path manually on Ubuntu machine

I installed the package like you did above:
conda install cookiecutter
Waited until the installation happened and then used:
cookiecutter https://github.com/drivendata/cookiecutter-data-science
I got the following output
project_name [project_name]: Fred
repo_name [Fred]: Fred
author_name [Your name (or your organization/company/team)]: Joseph
Cunningham
description [A short description of the project.]: test
Select open_source_license:
1 - MIT
2 - BSD-3-Clause
3 - No license file
Choose from 1, 2, 3 [1]: 3
s3_bucket [[OPTIONAL] your-bucket-for-syncing-data (do not include
's3://')]:
aws_profile [default]:
Select python_interpreter:
1 - python
2 - python3
Choose from 1, 2 [1]: 2
The people above are saying it is not a valid git but it worked for me.

I was behind a corporate firewall which prevented me from using
'cookiecutter https://github.com/drivendata/cookiecutter-data-science'
This may be an issue for someone else who lands up here.

Python 2.7 - grammar-check 1.3.1 library: HTTP Error 400: Bad Request

I am trying to use the grammar-check 1.3.1 library for Python
I have installed it using the pip commands
$ pip install --upgrade 3to2
$ pip install --upgrade language-check
I also unzipped the LanguageTool file from this link and moved it to C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check
Then I tried to run a test program
import grammar_check
tool = grammar_check.LanguageTool('en-GB')
text = 'This are bad.'
matches = tool.check(text)
len(matches)
But I got this error:
Traceback (most recent call last):
File "grammar-checker-test.py", line 2, in <module>
tool = grammar_check.LanguageTool('en-GB')
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 196, in __init__
self._language = LanguageTag(language)
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 405, in __new__
return unicode.__new__(cls, cls._normalize(tag))
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 426, in _normalize
for language in get_languages())
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 485, in get_languages
languages = LanguageTool._get_languages()
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 286, in _get_languages
for e in cls._get_root(url, num_tries=1):
File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\__init__.py", line 311, in _get_root
raise Error(u'{}: {}'.format(cls._url, e))
grammar_check.Error: http://127.0.0.1:8081: HTTP Error 400: Bad Request
How can I solve this error?

I had this error:
grammar_check.Error: http://127.0.0.1:8081:
I am not sure why you have to start server on your local machine as this piece of code does exactly that:
tool = grammar_check.LanguageTool('en-GB')
If you will have one more language_tool server running on 8081, then it will try to start server on port 8082 and if even that is busy on 8083 and then will fail (which was happening in my case).
Anyways for my error, I had to debug a lot and finally figured out that server was not able to run as my JAVA runtime environment was messed up.

The problem was that I wasn't running the LanguageTool server.
In order to do that I have to check if I have the LanguageTool folder (its version must be < 3.3, in my case it's the 2.2) inside my grammar_check folder.
Then using the cmd shell I have to move into this LanguageTool folder, which in my case is located in this path:
C:\Users\me\AppData\Roaming\Python\Python27\site-packages\grammar_check\LanguageTool-2.2
than, I have to run the server with this command
java -cp languagetool-server.jar org.languagetool.server.HTTPServer --port 8081
and voilà grammar_check is working well

Setting up the EB CLI - error nonetype get_frozen_credentials

Select a default region
1) us-east-1 : US East (N. Virginia)
2) us-west-1 : US West (N. California)
3) us-west-2 : US West (Oregon)
4) eu-west-1 : EU (Ireland)
5) eu-central-1 : EU (Frankfurt)
6) ap-southeast-1 : Asia Pacific (Singapore)
7) ap-southeast-2 : Asia Pacific (Sydney)
8) ap-northeast-1 : Asia Pacific (Tokyo)
9) ap-northeast-2 : Asia Pacific (Seoul)
10) sa-east-1 : South America (Sao Paulo)
11) cn-north-1 : China (Beijing)
(default is 3):5
When I choose a number or just leave it blank.. the following error appears:
ERROR: AttributeError :: 'NoneType' object has no attribute
'get_frozen_credentials'
after running eb init --debug:
Traceback (most recent call last): File "/usr/local/bin/eb", line 11,
in
sys.exit(main()) File "/Library/Python/2.7/site-packages/ebcli/core/ebcore.py", line 149, in
main
app.run() File "/Library/Python/2.7/site-packages/cement/core/foundation.py", line
694, in run
self.controller._dispatch()
File "/Library/Python/2.7/site-packages/cement/core/controller.py", line
455, in _dispatch
return func()
File "/Library/Python/2.7/site-packages/cement/core/controller.py", line
461, in _dispatch
return func()
File "/Library/Python/2.7/site-packages/ebcli/core/abstractcontroller.py",
line 57, in default
self.do_command()
File "/Library/Python/2.7/site-packages/ebcli/controllers/initialize.py",
line 67, in do_command
self.set_up_credentials()
File "/Library/Python/2.7/site-packages/ebcli/controllers/initialize.py",
line 152, in set_up_credentials
if not initializeops.credentials_are_valid():
File "/Library/Python/2.7/site-packages/ebcli/operations/initializeops.py",
line 24, in credentials_are_valid
elasticbeanstalk.get_available_solution_stacks()
File "/Library/Python/2.7/site-packages/ebcli/lib/elasticbeanstalk.py",
line 239, in get_available_solution_stacks
result = _make_api_call('list_available_solution_stacks')
File "/Library/Python/2.7/site-packages/ebcli/lib/elasticbeanstalk.py",
line 37, in _make_api_call
**operation_options)
File "/Library/Python/2.7/site-packages/ebcli/lib/aws.py", line 207, in make_api_call
response_data = operation(**operation_options)
File "/Library/Python/2.7/site-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Library/Python/2.7/site-packages/botocore/client.py", line 396, in _make_api_call
operation_model, request_dict)
File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 111, in make_request
return self._send_request(request_dict, operation_model)
File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 136, in _send_request
request = self.create_request(request_dict, operation_model)
File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 120, in create_request
operation_name=operation_model.name)
File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 226, in emit
return self._emit(event_name, kwargs)
File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 209, in _emit
response = handler(**kwargs)
File "/Library/Python/2.7/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/Library/Python/2.7/site-packages/botocore/signers.py", line 123, in sign
signature_version)
File "/Library/Python/2.7/site-packages/botocore/signers.py", line 153, in get_auth_instance
kwargs['credentials'] = self._credentials.get_frozen_credentials()
AttributeError: 'NoneType' object has no attribute 'get_frozen_credentials'

You got this error because you didn't initialize your AWS Access Key ID and AWS Secret Access Key
you should install first awscli by runing pip install awscli.
After you need to configure aws:
aws configure
After this you can run eb init

Ran into this error as well. Not sure of the root cause. But am still on Yosemite. When I tried AWS's install script
curl -s https://s3.amazonaws.com/elasticbeanstalk-cli-resources/install-ebcli.py | python
It downloaded EB CLI 3.7.3 (Python 2.7.1). A co-worker tried on his machine using homebrew and (since he hadn't run brew update in awhile) managed to get version 3.5.4 which worked perfectly.
Looks like Amazon may have broken something in the updates for Yosemite users so rather than wait I just downloaded the working version and had no issues running eb init
Here's the steps I took:
Amazon created a symlink to version 3.7 in usr/local/bin called eb. Remove this link rm /usr/local/bin/eb
cd /usr/local/Library
Use homebrew's brew log awsebcli to find the sha for the updated commit of version 3.5.4 (21e20f5).
Run git checkout 21e20f5 /usr/local/Library/Formula/aws-elasticbeanstalk.rb
Run brew install awsebcli and eb init should now run properly.
Hope that helps!

Hey I facing the same problem, running eb for the first time on Arch Linux.
What worked for me was instead of simply using
eb init
try a parameterized init, for example for interactive init
eb init -i
Reference: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-init.html
Hope that helps :)

I had the same problem.
I installed aws-cli ( brew install awsebcli )
Next, run "aws configure" and provide the correct details.
This solved my problem.

Was also had same issue but solved it some how. As I was doing it for first time I prefer to follow step by step documentation but lead to above issue but solved by following simple steps
$ mkdir HelloWorld
$ cd HelloWorld
$ eb init -p PHP
$ echo "Hello World" > index.html
$ eb create dev-env
$ eb open
In my case while following document steps I never asked for id/secret but with above commands it was possible.
Hope it work for you!

With credentials stored in ~/.aws/credentials and even after calling aws configure, I still had to use --profile profile_name with the command that was failing with the "no attribute 'get_frozen_credentials'" error. This fixed it for me.

I had the same problem but none of the above helped me. My problem was solved after creating an IAM user http://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html and generating new keys for it instead of using root and root keys and doing things the legacy way.

GitPython error when pulling from remote repository

I'm trying to pull from a remote repo in a local one with this code:
repo = git.Repo('/home/user/repo/')
o = repo.remotes.origin
try:
o.pull()
except:
logging.exception("oops:")
This fails miserably with the following traceback:
Traceback (most recent call last):
File "/home/user/my_site/app/app.py", line 58, in regen_logs
o.pull()
File "/home/user/my_site/venv/lib/python2.7/site-packages/git/remote.py", line 665, in pull
proc = self.repo.git.pull(self, refspec, with_extended_output=True, as_process=True, v=True, **kwargs)
File "/home/user/my_site/venv/lib/python2.7/site-packages/git/cmd.py", line 440, in <lambda>
return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
File "/home/user/my_site/venv/lib/python2.7/site-packages/git/cmd.py", line 834, in _call_process
return self.execute(make_call(), **_kwargs)
File "/home/user/my_site/venv/lib/python2.7/site-packages/git/cmd.py", line 576, in execute
raise GitCommandNotFound(str(err))
GitCommandNotFound: [Errno 2] No such file or directory
However, when doing the exact three commands (Repo, set origin, pull()) in an interactive session, it works just fine:
>>> import git
>>> repo = git.Repo('/home/user/repo')
>>> o = repo.remotes.origin
>>> o.pull()
[<git.remote.FetchInfo object at 0x242d2b8>]
I do have Git installed on the system:
$ rpm -q git
git-1.8.3.1-6.el7.x86_64
Any clues what I'm doing wrong here?

Found the culprit. Turns out gunicorn, which I used to start the app, clears out most environment variables, and defines a PATH pointing to the virtual environment's directory. Thus, the git executable could not be located anywhere. I fixed this by calling gunicorn with an extra option:
-e GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git

You need to set path to git before import git, like this:
import os
os.environ['GIT_PYTHON_GIT_EXECUTABLE'] = '/usr/local/bin/git'
from git import Repo