Top products from r/scrapy
We found 2 product mention on r/scrapy. We ranked the 1 resulting product by number of redditors who mentioned them. Here are the top 20.
1. Wild Rose Studio Paper Pack, 6 by 6-Inch, Letters From Paris
Sentiment score: 1
Number of reviews: 1
WILD ROSE STUDIO-Paper PackThis package contains thirty-six 6x6 inch single-sided sheets of paper (three each of twelve designs)Available in a variety of designs each sold separately
Show Reddit reviews
Yes, i've been stuck on this for hours, there is an alternative https://www.amazon.co.uk/gp/offer-listing/B00CSWQSAE/ref=dp_olp_new?ie=UTF8&condition=new , by this I can simply extract prime or not field but for Sales Rank it's not there, i've to make a request for that and it'll double the total requests, I've tried it as well, but the problem is when scrapy sent request from that extracted url, it mixed the prime fields extracted from above url and sales rank field from the url extracted from above url. Can you help me?
That's the code :
def parse(self, response):
logging.info(response.headers.get("X-Crawlera-Slave"))
global list
global salesrank11
list = AmazonItem()
prime = response.xpath("//i[@aria-label='Prime']")
isok = response.xpath("//span[@class='a-size-medium olpCondition a-text-bold']/text()").get()
status = 0
if "New" in isok:
status = 1
if not prime:
prime = "Not Prime"
else:
prime = "Prime"
list['Prime'] = prime
list['Url'] = response.url
list['Status'] = status
link = response.xpath("//div[@id='olpProductImage']/a/@href").get()
yield scrapy.Request(url=link,callback=self.parse_rank)
def parse_rank(self, response):
sales_rank1 = response.xpath("//tr[@id='SalesRank']/td[@class='value']/text()").get()
if sales_rank1 is None:
sales_rank11 = 'No Sales Rank Available'
else:
sales_rank11 = sales_rank1.replace('(','').replace('\n','')
sales_rank11= int(sales_rank11.split()[0].replace(",",''))
list['SalesRank'] = sales_rank11
yield list