Develop Reference

Python Bigquery create temp table

When I create temp table via python, an error throws
400 Use of CREATE TEMPORARY TABLE requires a script or session
How can I create a session?
from google.colab import auth
from google.cloud import bigquery
from google.colab import data_table
client = bigquery.Client(project=project, location = location)
client.query('''
create temp table t_acquisted_users as
select *
from table_a
limit 10
''').result()

You can create a session using the BigQuery API using the create_session parameter in a job config, for example:
job_config=bigquery.QueryJobConfig(create_session=True)
More details on this excellent article:
https://dev.to/stack-labs/bigquery-transactions-over-multiple-queries-with-sessions-2ll5

That's how I fix it in quick. Awaiting others provide a better answer
# create session
client0 = bigquery.Client(project=project, location=location)
job = client0.query(
"SELECT 1;", # a query can't fail
job_config=bigquery.QueryJobConfig(create_session=True)
)
session_id = job.session_info.session_id
job.result()
# set default session
client = bigquery.Client(project=project, location=location,
default_query_job_config=bigquery.QueryJobConfig(
connection_properties=[
bigquery.query.ConnectionProperty(
key="session_id", value=session_id
)
]
))

fastapi snowflake connection only pulling 1 record

I am trying to read data from snowflake database using FASTAPI. I was able to create the connection which is able to pull data from snowflake.
The issue which I am facing right now is that I am only getting 1 record (instead of 10 records).
I suspect I am not using correct keyword while returning the data. appreciate any help.
Here is my code :-
from fastapi import FastAPI
import snowflake.connector as sf
import configparser
username='username_value'
password='password_value'
account= 'account_value'
warehouse= 'test_wh'
database= 'test_db'
ctx=sf.connect(user=username,password=password,account=account,warehouse=warehouse,database=database)
app = FastAPI()
#app.get('/test API')
async def fetchdata():
cursor = ctx.cursor()
cursor.execute("USE WAREHOUSE test_WH ")
cursor.execute("USE DATABASE test_db")
cursor.execute("USE SCHEMA test_schema")
sql = cursor.execute ("SELECT DISTINCT ID,NAME,AGE,CITY FROM TEST_TABLE WHERE AGE > 60")
for data in sql:
return data

You use return in your inner for-loop. This will return the first row encountered.
If you want to return all rows as a list, you can probably do (I'm not familiar with the snowflake connector):
return list(data)
instead of the for-loop, or sql.fetchall().

Insert_one no such method exists # pymongo 3.7.2

I´m pretty new learning Python and all the containing stuff.
I tried to make my first little steps, installing a MongoDB (working) and connecting to it.
from pymongo import MongoClient
from pprint import pprint
from random import randint
client = MongoClient('localhost', 27017)
db = client.test
collection = db.users
user = {"id": 1, "username": "Test"}
user_id = collection.insert_one(user).inserted_id
print(user_id)
This is the complete code.
pymongo Version: 3.7.2 checked with:
pip freeze | grep pymongo
Output: pymongo==3.7.2
Python Version: 3.7.1
If I try to execute my tiny script, the following error occurs:
'Collection' object is not callable.
If you meant to call the 'insert_one' method on a 'Collection'
object it is
failing because no such method exists.
Where is my Fault?
A little research showed that in pymongo v2 the ".insert_one" is ".insert", but the 3.7.2 version is installed so I should (and must) use the ".insert.one", not the ".insert"

insert_one according with pymongo documentation exists, for server version >= 3.2...
the use is:
user = {'x': 1}
result = db.test.insert_one(user)
result.inserted_id
more complete explanation about insert_one:
>>> db.test.count_documents({'x': 1})
0
>>> result = db.test.insert_one({'x': 1})
>>> result.inserted_id
ObjectId('54f112defba522406c9cc208')
>>> db.test.find_one({'x': 1})
{u'x': 1, u'_id': ObjectId('54f112defba522406c9cc208')}
the following below, I executed and works fine:
# importing client mongo to make the connection
from pymongo import MongoClient
print("--- Exemplo pymongo Connection ---")
# Connection to MongoDB
client = MongoClient('localhost', 27017)
# Selection the Database
db = client.python
# Select the collection
collection = db.users
# Set up a document
user = {"id": 1, "username": "Test"}
# insert one document into selected document
result = collection.insert_one(user)
# Selection just one document from collection
#result = collection.find_one()
# removing the document inserted
collection.delete_one(user)
# print the inserted_id
print("inserted_id: ", result.inserted_id)
Pymongo Documentation

Python - How to parse and save JSON to MYSQL database

As the title indicates, how does one use python to elegantly access an API and parse and save the JSON contents onto a relational database (MYSQL) for later access?
Here, I saved the data onto a pandas object. But how do I create a mysql database, save the json contents onto it, and access the contents for later use?
# Libraries
import json, requests
import pandas as pd
from pandas.io.json import json_normalize
# Set URL
url = 'https://api-v2.themuse.com/jobs'
# For loop to
for i in range(100):
data = json.loads(requests.get(
url=url,
params={'page': i}
).text)['results']
data_norm = pd.read_json(json.dumps(data))

You create your Mysql table on your server using something like Mysql Workbench CE. then in python you do this. I wasnt sure if you want to use data in for loop or data_norm so for ease of use, here some functions. insertDb() can be put in your for loop, since data will be overwriten by itself in every iteration.
import MySQLdb
def dbconnect():
try:
db = MySQLdb.connect(
host='localhost',
user='root',
passwd='password',
db='nameofdb'
)
except Exception as e:
sys.exit("Can't connect to database")
return db
def insertDb():
try:
db = dbconnect()
cursor = db.cursor()
cursor.execute("""
INSERT INTO nameoftable(nameofcolumn) \
VALUES (%s) """, (data))
cursor.close()
except Exception as e:
print e

If this is merely for storage for processing later, kind of like a cache, a varchar field is enough. If however you need to retrieve some structured jdata, JSON field is what you need.

Retrieve data from pymongo

I ve got a mongo database and I want to retrieve all documents with a cursor operation like in mongodb java api. I want to retrieve all username of database based on this cursor iteration. My code is like the above:
import pymongo
from pymongo import MongoClient
client = MongoClient('...', 27017)
db = client.test_database
db = client['...']
collection = db.test_collection
collection = db["..."]
result = collection.find()
obj = next(result, None)
if obj:
username= obj['username']
print username
I want for the collection to print all the usernames.

Just loop over the results and print the username. There's no reason to play with next()
.
for obj in collection.find():
print obj['username']