Develop Reference

Using ArcGIS Python API, how can I share all items in a specific group from ArcGIS online with the public?

I'm completely new at programming. As an administrator, I need to update items in an ArcGIS Online group to be shared with everyone. I'm building the script in Jupyter notebook. I found the below link on arcig python api page, and I've been able to get the items from the group, but I don't know how to loop and update the share of each item in the group. https://developers.arcgis.com/python/guide/accessing-and-managing-groups/

You could do a content search for the items and you can use the share method. This is not tested but should work.
from arcgis.gis import GIS
ago_gis = GIS()
webmaps = ago_gis.content.search("My Web Map", item_type="Web Map")
for x in webmaps:
x.share(everyone=True)

I was looking to do something similar but found that I had to use the share method carefully. The share method has default values of False for the 'everyone' and 'org' parameters. If you do not specify them, they default to False or do not share with everyone / do not share with my org.
In my case, I was trying to share an item with a group I made so I used the share method and only passed it the group argument. While it shared it with the intended group, it used the default values of the optional arguments 'everyone' and 'org' which set the items to only be viewed by the 'owner'. This broke any public facing apps and maps which this item was a part of.
In order to modify only the group setting, I needed to call for the item's current sharing settings (using the shared_with property), modify them as needed, and pass them into the share method with the modifications. This allowed me to keep them shared with the public or my org, but also add them to the group I needed them added to.
Here's the code I used in my notebook:
item_sharing_status = item.shared_with
item.share(everyone = item_sharing_status['everyone'],
org = item_sharing_status['org'],
groups = item_sharing_status['groups'] + [backup_group.id])

You can also put the items to share in a list with :
gis.content.share_items(items=[item1, item2, item3], everyone=True, org=True)
link to the doc

filtering with a for or if loop

So I'm currently working on a project where I'm trying to improve the code.
Currently, I have this as my views.py
def home1(request):
if request.user.is_authenticated():
location = request.GET.get('location', request.user.profile.location)
users = User.objects.filter(profile__location=location)
print users
matchesper = Match.objects.get_matches_with_percent(request.user)
print matchesper
matches = [match for match in matchesper if match[0] in users][:20]
Currently, users gives me back a list of user that have the same location as the request.user and matchesper gives me a match percentage with all users. Then matches uses these two lists to give me back a list of users with their match percentage and that match with the request.users location
This works perfectly, however as soon the number of users using the website increases this will become very slow? I could add [:50] at the end of matchesper for example but this means you will never match with older users that have the same location as the request.user.
My question is, is there not a way to just create matches with matchesper for only the users that have the same location? Could I use an if statement before matchesper or a for loop?
I haven't written this code but I do understand it, however when trying to improve it I get very stuck, I hope my explanation and question makes sense.
Thank you for any help in advance I'm very stuck!

(I'm assuming you're using the matchmaker project.)
In Django, you can chain QuerySet methods. You'll notice that the models.py file you're working from defines both a MatchQuerySet and a MatchManager. You might also notice that get_matches_with_percent is only defined on the Manager, not the QuerySet.
This is a problem, but not an insurmountable one. One way around it is to modify which QuerySet our manager method actually works on on. We can do this by creating a new method that is basically a copy of get_matches_with_percent, but with some additional filtering.
class MatchManager(models.Manager):
[...]
def get_matches_with_percent_by_location(self, user, location=None):
if location is None:
location = user.profile.location
user_a = Q(user_a__profile__location=location)
user_b = Q(user_b__profile__location=location)
qs = self.get_queryset().filter(user_a | user_b).matches(user).order_by('-match_decimal')
matches = []
for match in qs:
if match.user_a == user:
items_wanted = [match.user_b, match.get_percent]
matches.append(items_wanted)
elif match.user_b == user:
items_wanted = [match.user_a, match.get_percent]
matches.append(items_wanted)
else:
pass
return matches
Note the use of repeated chaining in line 10! That's the magic.
Other notes:
Q objects are a way of doing complex queries, like multiple "OR" conditions.
An even better solution would factor out the elements that are common to get_matches_with_percent and get_matches_with_percent_by_location to keep the code "DRY", but this is good enough for now ;)
Be mindful of the fact that get_matches_with_percent returns a vanilla list instead of a Django QuerySet; it's a "terminal" method. Thus, you can't use any other QuerySet methods (like filter) after invoking get_matches_with_percent.

Google Directory API groups

I am trying to use Google's admin directory API (with Google's python library).
I am able to list the users on the directory just fine, using some code like this:
results = client.users().list(customer='my_customer').execute()
However, those results do not include which groups the users are a member of. Instead, it appears that once I have the list of users, I then have to make a call to get a list of groups:
results = client.groups().list(customer='my_customer').execute()
And then go through each group and call the "members" api to see which users are in a group:
results = client.members().list(groupKey='[group key]').execute()
Which means that I have to make a new request for every group.
It seems to be horribly inefficient. There has to be a better way than this, and I'm just missing it. What is it?

There is no better way than the one you described (yet?): Iterate over groups and get member list of each one.
I agree that is not the answer you want to read, but it is also the way we do maintenance over group members.

The following method should be more efficient, although not 100% great :
For each user, call the groups.list method and pass the userKey parameter to specify that you only want groups who have user X as a member.
I'm not a Python developper, but it should look like this :
results = client.groups().list(customer='my_customer',userKey='user#domain.com').execute()

For each user:
results = client.groups().list(userKey=user,pageToken=None).execute()
where 'user' is the user's primary/alias email address or ID
This will return a page of groups the user is a member of, see:
https://developers.google.com/admin-sdk/directory/v1/reference/groups/list

grabbing HTTP GET parameter from url using Box API in python

I am dealing with the Box.com API using python and am having some trouble automating a step in the authentication process.
I am able to supply my API key and client secret key to Box. Once Box.com accepts my login credentials, they supply me with an HTTP GET parameter like
'http://www.myapp.com/finish_box?code=my_code&'
I want to be able to read and store my_code using python. Any ideas? I am new to python and dealing with APIs.

This is actually a more robust question than it seems, as it exposes some useful functions with web dev in general. You're basically asking how to separate my_code in the string 'http://www.myapp.com/finish_box?code=my_code&'.
Well let's take it in bits and pieces. First of all, you know that you only really need the stuff after the question mark, right? I mean, you don't need to know what website you got it from (though that would be good to save, let's keep that in case we need it later), you just need to know what arguments are being passed back. Let's start with String.split():
>>> return_string = 'http://www.myapp.com/finish_box?code=my_code&'
>>> step1 = return_string.split('?')
["http://www.myapp.com/finish_box","code=my_code&"]
This will return a list to step1 containing two elements, "http://www.myapp.com/finish_box" and "code=my_code&". Well hell, we're there! Let's split the second one again on the equals sign!
>>> step2 = step1[1].split("=")
["code","my_code&"]
Well lookie there, we're almost done! However, this doesn't really allow any more robust uses of it. What if instead we're given:
>>> return_string = r'http://www.myapp.com/finish_box?code=my_code&junk_data=ohyestheresverymuch&my_birthday=nottoday&stackoverflow=usefulplaceforinfo'
Suddenly our plan doesn't work. Let's instead break that second set on the & sign, since that's what's separating the key:value pairs.
step2 = step1[1].split("&")
["code=my_code",
"junk_data=ohyestheresverymuch",
"my_birthday=nottoday",
"stackoverflow=usefulplaceforinfo"]
Now we're getting somewhere. Let's save those as a dict, shall we?
>>> list_those_args = []
>>> for each_item in step2:
>>> list_those_args[each_item.split("=")[0]] = each_item.split("=")[1]
Now we've got a dictionary in list_those_args that contains key and value for every argument the GET passed back to you! Science!
So how do you access it now?
>>> list_those_args['code']
my_code

You need a webserver and a cgi-script to do this. I have setup a single python script solution to this to run this. You can see my code at:
https://github.com/jkitchin/box-course/blob/master/box_course/cgi-bin/box-course-authenticate
When you access the script, it redirects you to box for authentication. After authentication, if "code" is in the incoming request, the code is grabbed and redirected to the site where tokens are granted.
You have to setup a .htaccess file to store your secret key and id.

How to implement full text search in Django?

I would like to implement a search function in a django blogging application. The status quo is that I have a list of strings supplied by the user and the queryset is narrowed down by each string to include only those objects that match the string.
See:
if request.method == "POST":
form = SearchForm(request.POST)
if form.is_valid():
posts = Post.objects.all()
for string in form.cleaned_data['query'].split():
posts = posts.filter(
Q(title__icontains=string) |
Q(text__icontains=string) |
Q(tags__name__exact=string)
)
return archive_index(request, queryset=posts, date_field='date')
Now, what if I didn't want do concatenate each word that is searched for by a logical AND but with a logical OR? How would I do that? Is there a way to do that with Django's own Queryset methods or does one have to fall back to raw SQL queries?
In general, is it a proper solution to do full text search like this or would you recommend using a search engine like Solr, Whoosh or Xapian. What are their benefits?

I suggest you to adopt a search engine.
We've used Haystack search, a modular search application for django supporting many search engines (Solr, Xapian, Whoosh, etc...)
Advantages:
Faster
perform search queries even without querying the database.
Highlight searched terms
"More like this" functionality
Spelling suggestions
Better ranking
etc...
Disadvantages:
Search Indexes can grow in size pretty fast
One of the best search engines (Solr) run as a Java servlet (Xapian does not)
We're pretty happy with this solution and it's pretty easy to implement.

Actually, the query you have posted does use OR rather than AND - you're using \ to separate the Q objects. AND would be &.
In general, I would highly recommend using a proper search engine. We have had good success with Haystack on top of Solr - Haystack manages all the Solr configuration, and exposes a nice API very similar to Django's own ORM.

Answer to your general question: Definitely use a proper application for this.
With your query, you always examine the whole content of the fields (title, text, tags). You gain no benefit from indexes, etc.
With a proper full text search engine (or whatever you call it), text (words) is (are) indexed every time you insert new records. So queries will be a lot faster especially when your database grows.

SOLR is very easy to setup and integrate with Django. Haystack makes it even simpler.

For full text search in Python, look at PyLucene. It allows for very complex queries. The main problem here is that you must find a way to tell your search engine which pages changed and update the index eventually.
Alternatively, you can use Google Sitemaps to tell Google to index your site faster and then embed a custom query field in your site. The advantage here is that you just need to tell Google the changed pages and Google will do all the hard work (indexing, parsing the queries, etc). On top of that, most people are used to use Google to search plus it will keep your site current in the global Google searches, too.

I think full text search on an application level is more a matter of what you have and how you expect it to scale. If you run a small site with low usage I think it might be more affordable to put some time into making an custom full text search rather than installing an application to perform the search for you. And application would create more dependency, maintenance and extra effort when storing data. By making your search yourself and you can build in nice custom features. Like for example, if your text exactly matches one title you can direct the user to that page instead of showing the results. Another would be to allow title: or author: prefixes to keywords.
Here is a method I've used for generating relevant search results from a web query.
import shlex
class WeightedGroup:
def __init__(self):
# using a dictionary will make the results not paginate
# but it will be a lot faster when storing data
self.data = {}
def list(self, max_len=0):
# returns a sorted list of the items with heaviest weight first
res = []
while len(self.data) != 0:
nominated_weight = 0
for item, weight in self.data.iteritems():
if weight > nominated_weight:
nominated = item
nominated_weight = weight
self.data.pop(nominated)
res.append(nominated)
if len(res) == max_len:
return res
return res
def append(self, weight, item):
if item in self.data:
self.data[item] += weight
else:
self.data[item] = weight
def search(searchtext):
candidates = WeightedGroup()
for arg in shlex.split(searchtext): # shlex understand quotes
# Search TITLE
# order by date so we get most recent posts
query = Post.objects.filter_by(title__icontains=arg).order_by('-date')
arg_hits = query.count() # count is cheap
if arg_hits > 1000:
continue # skip keywords which has too many hits
# Each of these are expensive as it would transfer data
# from the db and build a python object,
for post in query[:50]: # so we limit it to 50 for example
# more hits a keyword has the lesser it's relevant
candidates.append(100.0 / arg_hits, post.post_id)
# TODO add searchs for other areas
# Weight might also be adjusted with number of hits within the text
# or perhaps you can find other metrics to value an post higher,
# like number of views
# candidates can contain a lot of stuff now, show most relevant only
sorted_result = Post.objects.filter_by(post_id__in=candidates.list(20))