I want to pass two parameters in GET method in django. The one is start_date and the second is end_date. But I just know about passing only one method. here is my code where I want to filter out account deposit history in range of dates .
class BusinessDonationHistoryController(BaseController):
view = ListView
#entity(Business, arg="business")
def read(self, request, response, business):
request.query_params.get() #Here I need help
deposits = Deposit.objects.filter(
business=business, deposit_type=deposit_types.BUSINESS_DONATION)
credit_accounts = CreditAccount.objects.filter(deposit__in=deposits)
deposit_map = defaultdict(list)
# create a mapping of deposit id and associated credit accounts
for credit_account in credit_accounts:
deposit_map[credit_account.deposit_id].append(credit_account)
history = []
for deposit in deposits:
history.append({
"date": deposit.created_date,
"total_amount": from_cents(deposit.amount),
"amount_disbursed": from_cents(sum([ca.total_amount - ca.current_amount for ca in deposit_map.get(deposit.id)]))
})
print(history)
Checked all the links on StackOverflow but I found nothing relevant.
checked this link even
Related
I have been trying to scrape some data but keep getting a blank value or None. I've tried doing next sibling and failed (I probably did it wrong). Any and all help is greatly appreciated. Thank you in advance.
Website to scrape (final): https://www.unegui.mn/azhild-avna/ulan-bator/
Website to test (current, has less listings): https://www.unegui.mn/azhild-avna/mt-hariltsaa-holboo/slzhee-tehnik-hangamzh/ulan-bator/
Code Snippet:
def parse(self, response, **kwargs):
cards = response.xpath("//li[contains(#class,'announcement-container')]")
# parse details
for card in cards:
company = card.xpath(".//*[#class='announcement-block__company-name']/text()").extract_first()
date_block = card.xpath("normalize-space(.//div[contains(#class,'announcement-block__date')]/text())").extract_first().split(',')
date = date_block[0]
city = date_block[1]
item = {'date': date,
'city': city,
'company': company
}
HTML Snippet:
<div class="announcement-block__date">
<span class="announcement-block__company-name">Электро экспресс ХХК</span>
, Өчигдөр 13:05, Улаанбаатар</div>
Expected Output:
date = Өчигдөр 13:05
city = Улаанбаатар
UPDATE: I figured out how to get my date and city data. I ended up using follow next sibling to get date, split by comma, and get the 2nd and 3rd values.
date_block = card.xpath("normalize-space(.//div[contains(#class,'announcement-block__date')]/span/following-sibling::text())").extract_first().split(',')
date = date_block[1]
city = date_block[2]
Extra:
If anyone can tell me or refer me to how I can setup my pipeline file would be greatly appreciated. Is it correct to use pipeline or should you use items.py? Currently I have 3 spiders in the same project folder: apartments, jobs, cars. I need to clean my data and transform it. For example, for the jobs spider I am currently working on as shown above I want to create the following manipulations:
if salary is < 1000, then replace with string 'Negotiable'
if date contains the text "Өчигдөр" then replace with 'Yesterday'
without deleting the time
if employer contains value 'Хувь хүн' then change company value to 'Хувь хүн'
my pipelines.py file:
from itemadapter import ItemAdapter
class ScrapebooksPipeline:
def process_item(self, item, spider):
return item
my items.py file:
import scrapy
class ScrapebooksItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
pass
I changed your xpath to a smaller scope.
extract_first() will get the first instance, so use getall() instead.
In order to get the date I had to use regex (most of the results have time but not date so if you get a blank for the date it's perfectly fine).
I can't read the language so I had to guess (kind of) for the city, but even if it's wrong you can get the point.
import scrapy
import re
class TempSpider(scrapy.Spider):
name = 'temp_spider'
allowed_domains = ['unegui.mn']
start_urls = ['https://www.unegui.mn/azhild-avna/ulan-bator/']
def parse(self, response, **kwargs):
cards = response.xpath('//div[#class="announcement-block__date"]')
# parse details
for card in cards:
company = card.xpath('.//span/text()').get()
date_block = card.xpath('./text()').getall()
date = date_block[1].strip()
date = re.findall(r'(\d+-\d+-\d+)', date)
if date:
date = date[0]
else:
date = ''
city = date_block[1].split(',')[2].strip()
item = {'date': date,
'city': city,
'company': company
}
yield item
Output:
[scrapy.core.scraper] DEBUG: Scraped from <200 https://www.unegui.mn/azhild-avna/ulan-bator/>
{'date': '2021-11-07', 'city': 'Улаанбаатар', 'company': 'Arirang'}
[scrapy.core.scraper] DEBUG: Scraped from <200 https://www.unegui.mn/azhild-avna/ulan-bator/>
{'date': '2021-11-11', 'city': 'Улаанбаатар', 'company': 'Altangadas'}
[scrapy.core.scraper] DEBUG: Scraped from <200 https://www.unegui.mn/azhild-avna/ulan-bator/>
...
...
...
Looks like you are missing indentation.
Instead
def parse(self, response, **kwargs):
cards = response.xpath("//li[contains(#class,'announcement-container')]")
# parse details
for card in cards: date_block = card.xpath("normalize-space(.//div[contains(#class,'announcement-block__date')]/text())").extract_first().split(',')
date = date_block[0]
city = date_block[1]
Try this:
def parse(self, response, **kwargs):
cards = response.xpath("//li[contains(#class,'announcement-container')]")
# parse details
for card in cards: date_block = card.xpath("normalize-space(.//div[contains(#class,'announcement-block__date')]/text())").extract_first().split(',')
date = date_block[0]
city = date_block[1]
Working with SAP data, we are willing to enrich sales data with the last customer. Depending on the sales type, there are different partner function codes that correspond to the last company to which the sale is performed (e.g.: we may have indirect or direct sales). For now, we have been considering tables VBAP/VBAK/VBPA. We extract data from each table to separate files using sap4j, and then join VBAP and VBPA on VBELN, and consider partner codes WE (goods recipient) or custom consignation codes indicating the last buyer for consignations.
Is there some accurate way to know who is the last buyer in the chain for a given sale?
It can be done in the following way:
def sales_tabkey(row):
return "001{}{}".format(row['VBELN'], row['POSNR'])
def expected_partner_function_for_sales_type(row):
consignation_codes = set(['ORK', 'XKB', 'ZSOK', 'ZLZK', 'ZTSK', 'KE', 'ZED', 'ZZN'])
if row['AUART'] in consignation_codes:
return 'ZK'
return 'WE'
def get_kunnrf_frame(vbap, vbak, vbpa, kna):
consignation_codes = set(['ORK', 'XKB', 'ZSOK', 'ZLZK', 'ZTSK', 'KE', 'ZED', 'ZZN'])
df = pd.merge(vbap, vbak, on=['VBELN'], how='left')
df = pd.merge(df, vbpa, on='VBELN', how='left')
df["EXPPARVW"]=df.apply(expected_partner_function_for_sales_type, axis=1)
# KUNNR in kna is considered end_client_id
df = pd.merge(df, kna, on='ADRNR', how='left')[['VBELN','POSNR', 'KUNNR','end_client_id', 'ADRNR', 'PARVW', 'EXPPARVW', 'AUART']].drop_duplicates()
df['TABKEY']=df.apply(sales_tabkey,axis=1)
endclient_tabkeys = set(df.TABKEY.unique())
dfa = df[df.PARVW==df['EXPPARVW']]
dfb = df[df.TABKEY.isin(endclient_tabkeys.difference(set(dfa.TABKEY.unique())))]
return pd.concat([dfa, dfb])
I have a spreadsheet of members designed as below:
My aim is to upload some columns and exclude others. In this case, I wish to upload only the name, age and email and exclude the others. I have been able to achieve this using the slice method as shown below:
def load_imported_members
spreadsheet = open_spreadsheet
spreadsheet.default_sheet = 'Worksheet'
header = spreadsheet.row(1)
(2..spreadsheet.last_row).map do |i|
row = Hash[[header, spreadsheet.row(i)].transpose]
member = Member.find_by_id(row["id"]) || Member.new
member.attributes = row.to_hash.slice("id", "name", "age", "email")
member
end
end
The problem is that last_row considers all the rows upto the last one (13), and since there are validations on the form, there are errors due to missing data as a result of the empty rows (which shouldn’t be considered). Is there a way I can upload only specific columns as I have done, yet limit to only the rows that have data?
You might want to chain the map call off of a reject filter like this example
You may just need to change the map line to this (assuming the missing rows all look like those above):
(2..spreadsheet.last_row).reject{|i| spreadsheet.row(i)[0] }.map do |i|
This is assuming the blank rows return as nil and that blank rows will always have all four desired fields blank as shown in the image. The reject call tests to see if spreadsheet.row(i)[0], the id column, is nil, if so the item is rejected from the list output given to map
Thanks for this question. I have learned some things from this question.
I have shortlisted your answer [note: use 'ROO' gem]
def load_imported_members(member)
spreadsheet = open_spreadsheet(member)
spreadsheet.each do |records|
record = #spreadsheet ? Hash[[#header, #spreadsheet.row(records)].transpose] : Hash[records] # transpose for xlsx records and
attributes = {id: record['id'], name: record['name'], email: record['email'], age: record['age']}
member_object = Member.new(attributes)
if member_object.valid?
if Member.find(attributes[:id])
Member.find(attributes[:id]).update(attributes)
else
member_object.save
end
end
end
end
You can parse your uploaded file using Roo gem.
def self.open_spreadsheet(member)
case File.extname(member.file.original_filename)
when ".csv" then
Roo::CSV.new(member.file.expiring_url, csv_options: {headers: true, skip_blanks: true, header_converters: ->(header) { header.strip }, converters: ->(data) { data ? data.strip : nil }})
when ".xlsx", ".xls" then
#spreadsheet = Roo::Spreadsheet.open(member.file.expiring_url)
#header = #spreadsheet.row(1)
(2..#spreadsheet.last_row)
end
end
Here I have used s3 uploaded url i.e expiring_url. Hope This will helpful. I have not tested. sorry, for small errors.
If you have used validations for name, email, and age. This will surely help u..
i want to display cryptocurrency prices on my site. Therefor i parse the latest BTC/USD price from coinmarketcap.com
now i want to display them in a list but i first dont know who to save the symbol from the json to my database and second how can i display my view propperly. Currently i only save key:value of price_usd where key is the name of the currency.
views.py
def crypto_ticker(request):
list_prices = CryptoPrices.objects.get_queryset().order_by('-pk')
paginator = Paginator(list_prices, 100) # Show 100 prices per page
page = request.GET.get('page')
price = paginator.get_page(page)
return render(request, 'MyProject/crypto_ticker.html', {'price': price})
urls.py
url(r'^crypto_ticker/$', MyProject_views.crypto_ticker, name='crypto_ticker'),
models.py
class CryptoPrices(models.Model):
symbol = models.CharField(max_length=10)
key = models.CharField(max_length=30)
value = models.CharField(max_length=200)
celery update task:
#periodic_task(run_every=(crontab(minute='*/1')), name="Update Crypto rate(s)", ignore_result=True)
def get_exchange_rate():
api_url = "https://api.coinmarketcap.com/v1/ticker/?limit=100"
try:
exchange_rates = requests.get(api_url).json()
for exchange_rate in exchange_rates:
CryptoPrices.objects.update_or_create(key=exchange_rate['id'],
defaults={'value': round(float(exchange_rate['price_usd']), 3)}
)
logger.info("Exchange rate(s) updated successfully.")
except Exception as e:
print(e)
Surely just adding
symbol= exchange_rate['symbol']
to your update_or_create will work?
The JSON from coinmarketcap sets that as a key in the dictionary, unless you want an image that they use?
In that case you would have to save copies of that image yourself, create a mapping from the text of the symbol to the image itself, and format that on your html output.
Given a JSON document on couchbase, for example, a milestone collections, which is similar to this:
{
"milestoneDate" : /Date(1335191824495+0100)/,
"companyId" : 43,
"ownerUserId": 475,
"participants" : [
{
"userId": 2,
"docId" : "132546"
},
{
"userId": 67,
"docId" : "153"
}
]
}
If I were to select all the milestones of the company 43 and want to order them by latest first.. my view on couchbase would be something similar to this:
function (doc, meta) {
if(doc.companyId && doc.milestoneDate)
{
//key made up of date particles + company id
var eventKey = dateToArray(new Date(parseInt(doc.milestoneDate.substr(6))));
eventKey.push(doc.companyId);
emit(eventKey, null);
}
}
I do get both dates and the company Id on rest urls.. however, being quite new to couchbase, I am unable to work out how to restrict the view to return only milestones of company 43
The return key is similar to this:
"key":[2013,6,19,16,11,25,14]
where the last element (14) is the company id.. which is quite obviously wrong.
The query parameters that I have tried are:
&descending=true&startkey=[{},43]
&descending=true&startkey=[{},43]&endKey=[{},43]
tried adding companyId to value but couldn't restrict return results by value.
And according to couchbase documentation I need the date parts in the beginning to sort them. How do I restrict them by company id now, please?
thanks.
Put the company id at the start of the array, and because you'll be limiting by company id, couchbase sorts by company id and then by date array so you will be only ever getting the one company's milestone documents
I'd modify the view to emit
emit([doc.copmanyId, eventKey], null);
and then you can query the view with
&descending=true&startkey=[43,{}]
This was what worked for me previously..
I went back and tried it with end key and this seems to work - restricts and orders as required:
&descending=true&startkey=[43,{}]&endkey=[42,{}]
or
&descending=true&startkey=[43,{}]&endkey=[43,{}]&inclusive_end=true
either specify the next incremented/decremented value (based on descending flag) with end key, or use the same endkey as startkey and set inclusiveEnd to true
Both of these options should work fine. (I only tested the one with endkey=42 but they should both work)