Develop Reference

PowerShell script to get user information takes hours to run--any way to do it faster?

This PowerShell script to get a lot of user information from Microsoft 365 and write to csv works fine, but is taking a LONG time to run (like hours) for about 12k users. Is there any way to do this in a faster way? Is there an API I can call from Python that would do this faster?
# Get all users from AAD
$all = get-azureaduser -All $true
# Empty the output file we will use
$path = 'c:\blahblahpath\M365users.csv'
$path -replace ' ', '` '
Clear-Content -Path $path
# Loop through all users, get attributes and write to csv
$all | foreach-object {
$user = $_
$mbx = ""
$fwd = ""
$mbx = get-mailbox $user.userprincipalname -ErrorAction SilentlyContinue
$fwd = $mbx.forwardingsmtpaddress
$extem = $mbx.externalemailaddress
$aliases = $mbx.emailaddresses
# Expanding only the Extension Attributes related to the user and converting the Dictionary to Custom Object
# so that keys can be accessed through the dot (.) operator
$Extension_Attributes = New-Object Psobject -Property $user.ExtensionProperty
# Combining the required attributes from the user object and extension_attributes to A single object
$u_properties = [pscustomobject] #{
"UserPrincipalName" = $user.UserPrincipalName
"GivenName" = $user.GivenName
"Surname" = $user.Surname
"M3-DisplayName" = $user.DisplayName
"JobTitle" = $user.JobTitle
"Department" = $user.Department
"CompanyName" = $user.CompanyName
"M3-Mail" = $user.mail
"M3-OtherMails" = $user.OtherMails[0]
"ForwardingSmtpAddress" = $fwd
"ExternalEmailAddress" = $extem
"Aliases" = $aliases
"DeliverToMailboxAndForward" = $keepinmbx
"TelephoneNumber" = $user.TelephoneNumber
"PhysicalDeliveryOfficeName" = $user.PhysicalDeliveryOfficeName
"PreferredLanguage" = $user.PreferredLanguage
"MailNickName" = $user.MailNickName
"StreetAddress" = $user.StreetAddress
"City" = $user.City
"State" = $user.State
"PostalCode" = $user.PostalCode
"Country" = $user.Country
"UsageLocation" = $user.UsageLocation
"Mobile" = $user.Mobile
"UserType" = $user.UserType
"AccountEnabled" = $user.AccountEnabled
"LastLogon" = $user.LastLogon
"BlockCredential" = $user.BlockCredential
"AssignedLicenses" = $user.AssignedLicenses
"Created" = $Extension_Attributes.createdDateTime
"MemberNumber" = $Extension_Attributes.EmployeeID
"TokenExp" = $user.RefreshTokensValidFromDateTime
"OLDMemNumExt" = $Extension_Attributes.extension_60a1274a0a9d4344bd172d81b06d0f50_MemberNumber
"Source" = $Extension_Attributes.extension_06f8dc5952b048a1879af4e793844173_Source
"M3-MemType" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_MemType
"M3-Status" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_Status
"EmailTeams" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_EmailTeams
"M3-EmailCC" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_EmailCC
"M3-EmailPri" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_EmailPri
"Cert" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_Cert
"M3-ExpDate" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_ExpDate
"WGTitle" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_WGTitle
"ChapTitle" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_ChapTitle
"OtherTitle" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_LdrTitle
"Othercodes" = $extension_Attributes.extension_11b052b5eb8f4c77973a06876d25a6a4_Othercodes
"ChapCodes" = $extension_Attributes.extension_11b052b5eb8f4c77973a06876d25a6a4_ChapCodes
"WGcodes" = $extension_Attributes.extension_11b052b5eb8f4c77973a06876d25a6a4_WGcodes
"Allcodes" = $extension_Attributes. extension_a14f14664c8e457a856d24680b31c9d4_AllCodes
"Cmems" = $Extension_Attributes.extension_4adef2735ffb4a9dafc9a85188ca512d_Cmems
"M3-isnet" = $Extension_Attributes.extension_01816427922e40c9b55cf4364bc171ce_IsNet
"SipProxyAddress" = $user.SipProxyAddress
}
# check
#write-host "got $u_properties"
$u_properties | Export-csv -Path $path -NoTypeInformation -Force -Append
}#
I timed the $all = get-azureaduser -All $true at about 20 seconds, so obviously the time is in the loop--maybe the 'get-mailbox...' is slow?
I'm fairly novice at PowerShell so I can't think of a lot of other ways to try doing this.

I believe with simply putting Export-Csv as the last statement of your pipeline your script time will improve greatly. Appending to a Csv means that in each loop iteration Export-Csv has to open the FileStream, seek until the EOF, append the new line (a single object) and close the stream. This is a really expensive thing to do 12k times. Instead, if you put it as the last statement, no -Append is needed and the FileStream is kept opened until no more output is coming from your ForEach-Object.
Get-AzureADUser -All $true | ForEach-Object {
# same logic here
} | Export-Csv -Path $path -NoTypeInformation # here is the last statement, no append needed.
To put it into perspective how bad is -Append inside a loop, you can run the following test:
$test = #{
'With Append' = {
$tmp = New-TemporaryFile
0..12000 | ForEach-Object {
[pscustomobject]#{
foo = 'bar'
} | Export-Csv $tmp.FullName -Append
}
$tmp | Remove-Item
}
'Pipeline Processing' = {
$tmp = New-TemporaryFile
0..12000 | ForEach-Object {
[pscustomobject]#{
foo = 'bar'
}
} | Export-Csv $tmp.FullName
$tmp | Remove-Item
}
}
$test.GetEnumerator() | ForEach-Object {
[pscustomobject]#{
Test = $_.Key
Time = (Measure-Command { & $_.Value }).TotalMilliseconds
}
}

i want to use list in args of celery beat

i have list and in list i have 10 city i want to use all of my city to the celery beat but i cant
cityList = ["Tehran", "Shiraz", "Mashhad", "Qom", "Isfahan", "Ardabil", "Hamedan", "Yazd", "Tabriz", "Zavareh"]
app.conf.beat_schedule = {
'call_show_every_one_minute': {
"task": 'ali111.get_weather',
'schedule': crontab(minute='*/1'),
'args': ([cityList], ),
}
}
#app.task()
def get_weather(city):
con = redis.StrictRedis(host='localhost',port=6380, db=0, decode_responses=True)
appid = "b3bf68fdfc6ba46923cd50cb8b9a79c3"
URL = 'https://api.openweathermap.org/data/2.5/weather'
temp = con.get(city)
if temp is not None:
return temp
try:
PARAMS = {'q' :city, 'appid' :appid}
r = requests.get(url=URL, params=PARAMS)
city_temp = (r.json()['main']['temp']) - 273.15
my_temp = f"{round(city_temp,1)}c"
con.set(city, my_temp, ex=60)
return my_temp
except ConnectionError:
return "not internet connection"

Convert Python code to PHP/Laravel (Gate.io signature)

I am in the process of incorporating the gate,io rest api and am currently trying to convert the signature function from python to php(laravel).
Apparently there is a bug hiding in the conversion.
Can someone take a look and tell me if this is all correct or if something is missing here?
For improvement suggestions I would be grateful
Python code:
def gen_sign(method, url, query_string=None, payload_string=None):
key = '' # api_key
secret = '' # api_secret
t = time.time()
m = hashlib.sha512()
m.update((payload_string or "").encode('utf-8'))
hashed_payload = m.hexdigest()
s = '%s\n%s\n%s\n%s\n%s' % (method, url, query_string or "", hashed_payload, t)
sign = hmac.new(secret.encode('utf-8'), s.encode('utf-8'), hashlib.sha512).hexdigest()
return {'KEY': key, 'Timestamp': str(t), 'SIGN': sign}
Source: Gate.io API Signature string generation
Php Code:
public function createSignature($method, $url, $query=null, $payload=null, $algo = 'sha256'){
$key = 'xxx';
$secret= 'xxx';
$time = microtime(true);
$hashed_payload = hash_hmac($algo,$query ?? '');
$string = "{$methode}\n{$url}\n{$query ?? ''}\n{$hashed_payload}\n{$time}"
$sign = hash_hmac($algo,$string,$secret)
return ['KEY' => $key, 'Timestamp' => "{$time}", 'SIGN' => $sign]
}

i got the answer, i hope it will helps:
public function buildSignHeaders($method, $resourcePath, $queryParams = [], $payload = null)
{
$fullPath = parse_url(config('gate-io.host'), PHP_URL_PATH) . $resourcePath;
$fmt = "%s\n%s\n%s\n%s\n%s";
$timestamp = time();
$hashedPayload = hash("sha512", ($payload !== null) ? $payload : "");
$signatureString = sprintf(
$fmt,
$method,
$fullPath,
GuzzleHttp\Psr7\build_query($queryParams, false),
$hashedPayload,
$timestamp
);
$signature = hash_hmac("sha512", $signatureString, config('gate-io.apiSecretKey'));
return [
"KEY" => config('gate-io.apiKey'),
"SIGN" => $signature,
"Timestamp" => $timestamp
];
}

How to create dependent dropdowns using Jupyter Widgets to get data from dict?

I am trying to assign two variables with values using Jupyter Dropdowns where the first dropdown is a data center, and the second is a site available within this data center so that I can use these variables further in the code.
I tried multiple examples from different articles, but I cannot find what I am missing.
I have the following Dict:
data_center_environments = {
'US': {
'data_center': 'us..com',
'api_keys': {
'sandbox' : '3_EV',
'dev_parent' : '3_hK',
'stage' :'3_GE',
'prod' : '3_NL',
}
},
'RU': {
'data_center': 'ru..com',
'api_keys': {
'stage_parent' : '3_sN',
'prod_parent' : '3_R9',
}
},
'CN': {
'data_center': 'cn..cn',
'api_keys': {
'stage_parent' : '3_3k',
'prod_parent' : '3_MH',
}
},
'EU': {
'data_center': 'eu..com',
'api_keys': {
'sandbox' : '3_7h',
}
},
}
I created two functions to get datacenter and site:
def get_dc(dc_select=None):
dc = data_center_environments.get(dc_select)['data_center']
return dc
def get_site_api_key(dc_select=None, site_select=None):
site_api_key = data_center_environments[dc_select]['api_keys'][site_select]
return site_api_key
Here I describe two dropdowns:
dc_s = widgets.Dropdown(
options = data_center_environments.keys(),
description = 'Data Center:',
disabled = False,
)
site_s = widgets.Dropdown(
options=list(data_center_environments[dc_s.value]['api_keys']),
description = 'API Key:',
disabled = False,
)
def on_value_change(change):
dc = change.new
site_s.options = data_center_environments[dc_s.value]['api_keys']
dc_s.observe(on_value_change, 'value')
This is how I invoke them on a Jupyter Notebook page:
domain = interactive(get_dc, dc_select = dc_s)
site = interactive(get_site_api_key, dc_select = dc_s, site_select = site_s)
display(domain)
display(site)
Issues:
0. I have 3 dropdowns instead of two
1. I am getting an exception when I change datacenter value
2. When I am trying to print "domain", "domain.value" I am getting "None" as an output
What I am trying to achieve:
In:
domain =
site =
print(domain, site)
Out:
Select Datacenter [Drop Down: 'US', 'CN', 'RU', etc] -> pick 'US'
Select Site [Dropdown 'US': 'prod', 'stage', 'dev_parent', 'sandbox'] -> select 'prod'
'us..com' , '3_NL'
What am I doing wrong? How to change my code to make it work?
Thank you!

I end up writing a function that returns a dict, and I just get values from it.
The code below is a textbook example from the Widgets Guide.
Solution:
dc = 'US'
domain = widgets.Dropdown(
options = list(data_center_environments),
description = 'Data Center:',
disabled = False,
)
site = widgets.Dropdown(
options=list(data_center_environments[dc]['api_keys']),
description = 'API Key:',
disabled = False,
)
def on_value_change(change):
dc = change.new
site.options = data_center_environments[dc]['api_keys']
domain.observe(on_value_change, 'value')
def creds(data_center, api_key, use_secret):
data_center = data_center_environments[domain.value]['data_center']
api_key = site.value
creds = dict()
creds['data_center'] = data_center
creds['api_key'] = api_key
return creds

Python utility for parsing blocks?

I have a file that starts something like:
databaseCons = {
main = {
database = "readable_name",
hostname = "hostdb1.serv.com",
instances= {
slaves = {
conns = "8"
}
}
maxconns = "5",
user = "user",
pass = "pass"
}
}
So, what I'd like to do is parse this out into a dict of sub-dicts, something like:
{'main': {'database': 'readable_name', 'hostname': 'hostdb1.serv.com', 'maxconns': '5', 'instances': {'slave': {'maxCount': '8'}}, 'user': 'user', 'pass': 'pass'}}
I think the above makes sense... but please feel free to edit this if it doesn't. Basically I want the equivalent of:
conns = '8'
slave = dict()
slave['maxCount'] = conns
instances = dict()
instances['slave'] = slave
database = 'readable_name'
hostname = 'hostdb1.serv.com'
maxconns = '5'
user = 'user'
pas = 'pass'
main = dict()
main['database'] = database
main['hostname'] = hostname
main['instances'] = instances
main['maxconns'] = maxconns
main['user'] = user
main['pass'] = pas
databaseCons = dict()
databaseCons['main'] = main
Are there any modules out there that can handle this sort of parsing? Even what I've suggested above looks messy.. there's got to be a better way I'd imagine.

Here is a pyparsing parser for your config file:
from pyparsing import *
def to_dict(t):
return {k:v for k,v in t}
series = Forward()
struct = Suppress('{') + series + Suppress('}')
value = quotedString.setParseAction(removeQuotes) | struct
token = Word(alphanums)
assignment = Group(token + Suppress('=') + value + Suppress(Optional(",")))
series << ZeroOrMore(assignment).setParseAction(to_dict)
language = series + stringEnd
def config_file_to_dict(filename):
return language.parseFile(filename)[0]
if __name__=="__main__":
from pprint import pprint
pprint(config_file_to_dict('config.txt'))