we're using Google's Firestore for embedded machine configuration data. Because this data controls a configurable pageflow and lots of other things, it's segmented up into lots of subcollections. Each machine has it's own top level document in this system. However, it takes forever when we go to add machines to the fleet because we have to manually copy over all this data in multiple documents. Does anyone know how to go about recursively copying a Firestore document, all it's subcollections, their documents, subcollections, etc in Python. You'd have a document ref to the top level as well as a name for the new top level doc.
You can use something like this to recursively read and write from a collection to another one:
def read_recursive(
source: firestore.CollectionReference,
target: firestore.CollectionReference,
batch: firestore.WriteBatch,
) -> None:
global batch_nr
for source_doc_ref in source:
document_data = source_doc_ref.get().to_dict()
target_doc_ref = target.document(source_doc_ref.id)
if batch_nr == 500:
log.info("commiting %s batched operations..." % batch_nr)
batch.commit()
batch_nr = 0
batch.set(
reference=target_doc_ref,
document_data=document_data,
merge=False,
)
batch_nr += 1
for source_coll_ref in source_doc_ref.collections():
target_coll_ref = target_doc_ref.collection(source_coll_ref.id)
read_recursive(
source=source_coll_ref.list_documents(),
target=target_coll_ref,
batch=batch,
)
batch = db_client.batch()
read_recursive(
source=db_client.collection("src_collection_name"),
target=db_client.collection("target_collection_name"),
batch=batch,
)
batch.commit()
Writes are in batches and this saves a lot of time (in my case it finished in half the time compared with set).
The questions asks for Python, but in my case I needed to do recursive deep copy of Firestore docs / collections in NodeJS (Typescript), and using a Document as starting point of the recursion.
(This is a solution based on the Python script by #cristi)
Function definition
import {
CollectionReference,
DocumentReference,
DocumentSnapshot,
QueryDocumentSnapshot,
WriteBatch,
} from 'firebase-admin/firestore';
interface FirestoreCopyRecursiveContext {
batchSize: number;
/**
* Wrapped Firestore WriteBatch. In firebase-admin#11.0.1, you can't continue
* using the WriteBatch object after you call WriteBatch.commit().
*
* Hence, we need to replaced "used up" WriteBatch's with new ones.
* We also need to reset the count after committing, and because we
* want all recursive invocations to share the same count + WriteBatch instance,
* we pass this data via object reference.
*/
writeBatch: {
writeBatch: WriteBatch,
/** Num of items in current batch. Reset to 0 when `commitBatch` commits. */
count: number;
};
/**
* Function that commits the batch if it reached the limit or is forced to.
* The WriteBatch instance is automatically replaced with fresh one
* if commit did happen.
*/
commitBatch: (force?: boolean) => Promise<void>;
/** Callback to insert custom logic / write operations when we encounter a document */
onDocument?: (
sourceDoc: QueryDocumentSnapshot | DocumentSnapshot,
targetDocRef: DocumentReference,
context: FirestoreCopyRecursiveContext
) => unknown;
/** Callback to insert custom logic / write operations when we encounter a collection */
onCollection?: (
sourceDoc: CollectionReference,
targetDocRef: CollectionReference,
context: FirestoreCopyRecursiveContext
) => unknown;
logger?: Console['info'];
}
type FirestoreCopyRecursiveOptions = Partial<Omit<FirestoreCopyRecursiveContext, 'commitBatch'>>;
/**
* Copy all data from one document to another, including
* all subcollections and documents within them, etc.
*/
export const firestoreCopyDocRecursive = async (
/** Source Firestore Document Snapshot, descendants of which we want to copy */
sourceDoc: QueryDocumentSnapshot | DocumentSnapshot,
/** Destination Firestore Document Ref */
targetDocRef: DocumentReference,
options?: FirestoreCopyRecursiveOptions,
) => {
const batchSize = options?.batchSize ?? 500;
const writeBatchRef = options?.writeBatch || { writeBatch: firebaseFirestore.batch(), count: 0 };
const onDocument = options?.onDocument;
const onCollection = options?.onCollection;
const logger = options?.logger || console.info;
const commitBatch = async (force?: boolean) => {
// Commit batch only if size limit hit or forced
if (writeBatchRef.count < batchSize && !force) return;
logger(`Commiting ${writeBatchRef.count} batched operations...`);
await writeBatchRef.writeBatch.commit();
// Once we commit the batched data, we have to create another WriteBatch,
// otherwise we get error:
// "Cannot modify a WriteBatch that has been committed."
// See https://dev.to/wceolin/cannot-modify-a-writebatch-that-has-been-committed-265f
writeBatchRef.writeBatch = firebaseFirestore.batch();
writeBatchRef.count = 0;
};
const context = {
batchSize,
writeBatch: writeBatchRef,
onDocument,
onCollection,
commitBatch,
};
// Copy the contents of the current docs
const sourceDocData = sourceDoc.data();
await writeBatchRef.writeBatch.set(targetDocRef, sourceDocData, { merge: false });
writeBatchRef.count += 1;
await commitBatch();
// Allow to make additional changes to the target document from
// outside the func after copy command is enqueued / commited.
await onDocument?.(sourceDoc, targetDocRef, context);
// And try to commit in case user updated the count but forgot to commit
await commitBatch();
// Check for subcollections and docs within them
for (const sourceSubcoll of await sourceDoc.ref.listCollections()) {
const targetSubcoll = targetDocRef.collection(sourceSubcoll.id);
// Allow to make additional changes to the target collection from
// outside the func after copy command is enqueued / commited.
await onCollection?.(sourceSubcoll, targetSubcoll, context);
// And try to commit in case user updated the count but forgot to commit
await commitBatch();
for (const sourceSubcollDoc of (await sourceSubcoll.get()).docs) {
const targetSubcollDocRef = targetSubcoll.doc(sourceSubcollDoc.id);
await firestoreCopyDocRecursive(sourceSubcollDoc, targetSubcollDocRef, context);
}
}
// Commit all remaining operations
return commitBatch(true);
};
How to use it
const sourceDocRef = getYourFaveFirestoreDocRef(x);
const sourceDoc = await sourceDocRef.get();
const targetDocRef = getYourFaveFirestoreDocRef(y);
// Copy firestore resources
await firestoreCopyDocRecursive(sourceDoc, targetDocRef, {
logger,
// Note: In my case some docs had their doc ID also copied as a field.
// Because the copied documents get a new doc ID, we need to update
// those fields too.
onDocument: async (sourceDoc, targetDocRef, context) => {
const someDocPattern = /^nameOfCollection\/[^/]+?$/;
const subcollDocPattern = /^nameOfCollection\/[^/]+?\/nameOfSubcoll\/[^/]+?$/;
// Update the field that holds the document ID
if (targetDocRef.path.match(someDocPattern)) {
const docId = targetDocRef.id;
context.writeBatch.writeBatch.set(targetDocRef, { docId }, { merge: true });
context.writeBatch.count += 1;
await context.commitBatch();
return;
}
// In a subcollection, I had to update multiple ID fields
if (targetDocRef.path.match(subcollDocPattern)) {
const docId = targetDocRef.parent.parent?.id;
const subcolDocId = targetDocRef.id;
context.writeBatch.writeBatch.set(targetDocRef, { docId, subcolDocId }, { merge: true });
context.writeBatch.count += 1;
await context.commitBatch();
return;
}
},
});
Related
I am having an issue with the Fund me part of this course (16 hour video on YT) for brownie.
I am following Patrick and have the same set-up yet when I try to run this script I get an error he doesn't have:
brownie run scripts/deploy.py --network rinkeby
Brownie v1.17.2 - Python development framework for Ethereum
FundmeProject is the active project.
Running 'scripts/deploy.py::main'...
File "brownie/_cli/run.py", line 50, in main
return_value, frame = run(
File "brownie/project/scripts.py", line 103, in run
return_value = f_locals[method_name](*args, **kwargs)
File "./scripts/deploy.py", line 12, in main
deploy_fund_me()
File "./scripts/deploy.py", line 7, in deploy_fund_me
fund_me = FundMe.deploy({"from:account"}, publish_source=True)
File "brownie/network/contract.py", line 523, in _call_
raise AttributeError(
AttributeError: Final argument must be a dict of transaction parameters that includes a `from` field specifying the address to deploy from
Edit: needed more background info
The contract
// SPDX-License-Identifier: MIT
// Smart contract that lets anyone deposit ETH into the contract
// Only the owner of the contract can withdraw the ETH
pragma solidity ^0.6.6;
// Get the latest ETH/USD price from chainlink price feed
import "#chainlink/contracts/src/v0.6/interfaces/AggregatorV3Interface.sol";
import "#chainlink/contracts/src/v0.6/vendor/SafeMathChainlink.sol";
contract FundMe {
// safe math library check uint256 for integer overflows
using SafeMathChainlink for uint256;
//mapping to store which address depositeded how much ETH
mapping(address => uint256) public addressToAmountFunded;
// array of addresses who deposited
address[] public funders;
//address of the owner (who deployed the contract)
address public owner;
// the first person to deploy the contract is
// the owner
constructor() public {
owner = msg.sender;
}
function fund() public payable {
// 18 digit number to be compared with donated amount
uint256 minimumUSD = 50 * 10**18;
//is the donated amount less than 50USD?
require(
getConversionRate(msg.value) >= minimumUSD,
"You need to spend more ETH!"
);
//if not, add to mapping and funders array
addressToAmountFunded[msg.sender] += msg.value;
funders.push(msg.sender);
}
//function to get the version of the chainlink pricefeed
function getVersion() public view returns (uint256) {
AggregatorV3Interface priceFeed = AggregatorV3Interface(
0x8A753747A1Fa494EC906cE90E9f37563A8AF630e
);
return priceFeed.version();
}
function getPrice() public view returns (uint256) {
AggregatorV3Interface priceFeed = AggregatorV3Interface(
0x8A753747A1Fa494EC906cE90E9f37563A8AF630e
);
(, int256 answer, , , ) = priceFeed.latestRoundData();
// ETH/USD rate in 18 digit
return uint256(answer * 10000000000);
}
// 1000000000
function getConversionRate(uint256 ethAmount)
public
view
returns (uint256)
{
uint256 ethPrice = getPrice();
uint256 ethAmountInUsd = (ethPrice * ethAmount) / 1000000000000000000;
// the actual ETH/USD conversation rate, after adjusting the extra 0s.
return ethAmountInUsd;
}
//modifier: https://medium.com/coinmonks/solidity-tutorial-all-about-modifiers-a86cf81c14cb
modifier onlyOwner() {
//is the message sender owner of the contract?
require(msg.sender == owner);
_;
}
// onlyOwner modifer will first check the condition inside it
// and
// if true, withdraw function will be executed
function withdraw() public payable onlyOwner {
// If you are using version eight (v0.8) of chainlink aggregator interface,
// you will need to change the code below to
// payable(msg.sender).transfer(address(this).balance);
msg.sender.transfer(address(this).balance);
//iterate through all the mappings and make them 0
//since all the deposited amount has been withdrawn
for (
uint256 funderIndex = 0;
funderIndex < funders.length;
funderIndex++
) {
address funder = funders[funderIndex];
addressToAmountFunded[funder] = 0;
}
//funders array will be initialized to 0
funders = new address[](0);
}
}
Brownie-config.yaml:
dependencies:
# - <orghanization/repo>#<version>
- smartcontractkit/chainlink-brownie-contracts#1.1.1
compiler:
solc:
remappings:
- "#chainlink=smartcontractkit/chainlink-brownie-contracts#1.1.1"
dotenv: .env
wallets:
from_key: ${PRIVATE_KEY}
deploy.py
from brownie import FundMe
from scripts.helpful_scripts import get_account
def deploy_fund_me():
account = get_account()
fund_me = FundMe.deploy({"from:account"}, publish_source=True)
print(f"Contract deployed to {fund_me.address}")
def main():
deploy_fund_me(
helpful_scripts.py
from brownie import network, config, accounts
def get_account():
if network.show_active() == "development":
return accounts[0]
else:
return accounts.add(config["wallets"]["from_key"])
.env
export PRIVATE_KEY=0x******f5a557bbb30bb35f8c9929ded41eb9a15******b066d72b44890******
export WEB3_INFURA_PROJECT_ID=97417cf50bab449c88c09debfe******
export ETHERSCAN_TOKEN=ZMFY1FUWR67X4RZYHHGR6S4NNX1G******
I am not sure that this solve your issue, but you must correct your code:{"from:account"} ---> {"from": account}
I had the same issue with the upgrades project and adding this part to brownie-config.yaml resolve for me:
networks:
rinkeby:
verify: True
I hope that it is appropriate to ask whether someone with NodeJS experience can convert this code to python. I have already translated some small tiny bits, but I am getting errors when I run my code, I have tried for hours but had no luck in figuring out what my issue is. I have no NodeJs experience for clarification. Many Thanks in advance if do it :)
require('dotenv').config() // Load .env file
const axios = require('axios')
const Discord = require('discord.js')
const client = new Discord.Client()
function getPrices() {
// API for price data.
axios.get(`https://api.coingecko.com/api/v3/coins/markets?vs_currency=${process.env.PREFERRED_CURRENCY}&ids=${process.env.COIN_ID}`).then(res => {
// If we got a valid response
if(res.data && res.data[0].current_price && res.data[0].price_change_percentage_24h) {
let currentPrice = res.data[0].current_price || 0 // Default to zero
let priceChange = res.data[0].price_change_percentage_24h || 0 // Default to zero
let symbol = res.data[0].symbol || '?'
client.user.setPresence({
game: {
// Example: "Watching -5,52% | BTC"
name: `${priceChange.toFixed(2)}% | ${symbol.toUpperCase()}`,
type: 3 // Use activity type 3 which is "Watching"
}
})
client.guilds.find(guild => guild.id === process.env.SERVER_ID).me.setNickname(`${(currentPrice).toLocaleString().replace(/,/g,process.env.THOUSAND_SEPARATOR)}${process.env.CURRENCY_SYMBOL}`)
console.log('Updated price to', currentPrice)
}
else
console.log('Could not load player count data for', process.env.COIN_ID)
}).catch(err => console.log('Error at api.coingecko.com data:', err))
}
// Runs when client connects to Discord.
client.on('ready', () => {
console.log('Logged in as', client.user.tag)
getPrices() // Ping server once on startup
// Ping the server and set the new status message every x minutes. (Minimum of 1 minute)
setInterval(getPrices, Math.max(1, process.env.MC_PING_FREQUENCY || 1) * 60 * 1000)
})
// Login to Discord
client.login(process.env.DISCORD_TOKEN)
I'm trying to retrieve data programmatically through websockets and am failing due to my limited knowledge around this. On visiting the site at https://www.tradingview.com/chart/?symbol=ASX:RIO I notice one of the websocket messages being sent out is ~m~60~m~{"m":"quote_fast_symbols","p":["qs_p089dyse9tcu","ASX:RIO"]}
My code is as follows:
from websocket import create_connection
import json
ws = create_connection("wss://data.tradingview.com/socket.io/websocket?from=chart%2Fg0l68xay%2F&date=2019_05_27-12_19")
ws.send(json.dumps({"m":"quote_fast_symbols","p"["qs_p089dyse9tcu","ASX:RIO"]}))
result = ws.recv()
print(result)
ws.close()
Result of the print:
~m~302~m~{"session_id":"<0.25981.2547>_nyc2-charts-3-webchart-5#nyc2-compute-3_x","timestamp":1558976872,"release":"registry:5000/tvbs_release/webchart:release_201-106","studies_metadata_hash":"888cd442d24cef23a176f3b4584ebf48285fc1cd","protocol":"json","javastudies":"javastudies-3.44_955","auth_scheme_vsn":2}
I get this result no matter what message I send out, out of the almost multitude of messages that seem to be sent out. I was hoping one of the messages sent back will be the prices info for the low and highs for RIO. Is there other steps I should include to get this data? I understand there might be some form of authorisation needed but I dont know the workflow.
Yes, there is much more to setup and it needs to be done in order. The following example written in Node.js will subscribe to the BINANCE:BTCUSDT real time data and fetch historical 5000 bars on the daily chart.
Ensure you have proper value of the origin field set in header section before connecting. Otherwise your connection request will be rejected by the proxy. I most common ws there is no way to do this. Use faye-websocket instead
const WebSocket = require('faye-websocket')
const ws = new WebSocket.Client('wss://data.tradingview.com/socket.io/websocket', [], {
headers: { 'Origin': 'https://data.tradingview.com' }
});
After connecting you need to setup your data stream. I don't know if all of this commands needs to be performed. This probably can be shrink even more but it works. Basically what you need to do is to create new quote and chart sessions and within these sessions request stream of the data of the prior resolved symbol.
ws.on('open', () => {
const quote_session = 'qs_' + getRandomToken()
const chart_session = 'cs_' + getRandomToken()
const symbol = 'BINANCE:BTCUSDT'
const timeframe = '1D'
const bars = 5000
sendMsg(ws, "set_auth_token", ["unauthorized_user_token"])
sendMsg(ws, "chart_create_session", [chart_session, ""])
sendMsg(ws, "quote_create_session", [quote_session])
sendMsg(ws, "quote_set_fields", [quote_session,"ch","chp","current_session","description","local_description","language","exchange","fractional","is_tradable","lp","lp_time","minmov","minmove2","original_name","pricescale","pro_name","short_name","type","update_mode","volume","currency_code","rchp","rtc"])
sendMsg(ws, "quote_add_symbols",[quote_session, symbol, {"flags":['force_permission']}])
sendMsg(ws, "quote_fast_symbols", [quote_session, symbol])
sendMsg(ws, "resolve_symbol", [chart_session,"symbol_1","={\"symbol\":\""+symbol+"\",\"adjustment\":\"splits\",\"session\":\"extended\"}"])
sendMsg(ws, "create_series", [chart_session, "s1", "s1", "symbol_1", timeframe, bars])
});
ws.on('message', (msg) => { console.log(`RX: ${msg.data}`) })
And finally implementation of the helper methods
const getRandomToken = (stringLength=12) => {
characters = 'abcdefghijklmnopqrstuvwxyz0123456789'
const charactersLength = characters.length;
let result = ''
for ( var i = 0; i < stringLength; i++ ) {
result += characters.charAt(Math.floor(Math.random() * charactersLength))
}
return result
}
const createMsg = (msg_name, paramsList) => {
const msg_str = JSON.stringify({ m: msg_name, p: paramsList })
return `~m~${msg_str.length}~m~${msg_str}`
}
const sendMsg = (ws, msg_name, paramsList) => {
const msg = createMsg(msg_name, paramsList)
console.log(`TX: ${msg}`)
ws.send(createMsg(msg_name, paramsList))
}
I want to register the hotkey
So, I added key types
key_types.h
static const KeyID kKeyHangul = 0xEF31; /* Zenkaku/Hankaku */ //asmera modify to 0xef31 from 0xef26
key_types.cpp
keyNameMap[][]
{ "Hangul", kKeyHangul }
MSWindowsKeyState.cpp
/* 0x015 */ { kKeyHangul },
/* 0x115 */ { kKeyHangul },
and config script is
section: screens
server:
client1:
end
section: aliases
server:
YOO-Lab
client1:
YOO-SURFACE
end
section: links
# server:
# down = client1
# client:
# up = server
end
section: options
screenSaverSync = false
keystroke(control+1) = switchToScreen(server)
keystroke(control+2) = switchToScreen(client1)
#?? keystroke(ctrl+alt+space) = Hangul
end
I want to know how to convert (Ctrl+Alt+Space) to (Hangul)key
This works for me - Control+Alt+Space
Check out this reference for more keys: https://github.com/symless/synergy/blob/master/src/lib/synergy/key_types.cpp
is it possible to get all attributes for a particular node in pyqt ?
for example .. consider for following node:
< asset Name="3dAsset" ID="5"/>
i want to retrieve the ("Name" and "ID") strings
is it possible?
thanks in advance
You can retrieve the particular value of the attribute using the function,
QString QDomElement::attribute ( const QString & name, const QString & defValue = QString() ) const
To get all the attributes use,
QDomNamedNodeMap QDomElement::attributes () const
and you have to traverse through the DomNamedNodeMap and get the value of each of the attributes. Hope it helps.
Edit : Try this one.
With the QDomNamedNodeMap you are having give,
QDomNode QDomNamedNodeMap::item ( int index ) const
which will return a QDomNode for the particular attribute.
Then give,
QDomAttr QDomNode::toAttr () const
With the QDomAttr obtained give,
QString name () const
which will return the name of the attribute.
Hope it helps.
How to get first attribute name/value in PySide/PyQt:
if node.hasAttributes():
nodeAttributes = node.attributes()
attributeItem = nodeAttributes.item(0) #pulls out first item
attribute = attributeItem.toAttr()
attributeName = attr.name()
attributeValue = attr.value()
This just shows how to get one name/value pair, but it should be easy enough to extend looping with nodeAttributes.length() or something similar.
This is for c++. I Ran into the same problem. You need to convert to QDomAttr. I'm sure API is the same in python.
if( Node.hasAttributes() )
{
QDomNamedNodeMap map = Node.attributes();
for( int i = 0 ; i < map.length() ; ++i )
{
if(!(map.item(i).isNull()))
{
QDomNode debug = map.item(i);
QDomAttr attr = debug.toAttr();
if(!attr.isNull())
{
cout << attr.value().toStdString();
cout << attr.name().toStdString();
}
}
}