Scrapy Archives

This is a snippet of Python’s Scrapy code that I recently wrote to crawl and scrape job listings from a Nepal’s leading job listing site MeroJob.

I am planning to use data scraped from this site to perform some data analysis on the current job market trend of Nepal.

def parseJobLink(self, response):

        item = MerojobItem()

        item['employer'] = response.xpath('//span[@itemprop="name"]/text()').extract_first()

        employerDesc = response.xpath('//div[@itemprop="description"]//text()').extract()

        item['employerDesc'] = response.xpath('//div[@itemprop="description"]//text()').extract_first().encode('ascii', 'ignore').decode('ascii').strip()

        employerDesc = response.xpath('//div[@itemprop="description"]//text()').extract()

        item['employerDesc'] = "".join(x.encode('ascii', 'ignore').decode('ascii').strip() for x in employerDesc)

        item['jobTitle'] = response.xpath('//h1[@itemprop="title"]/text()').extract_first()

        item['jobViews'] = response.xpath('//span[@class="text-success"]/text()').extract_first()

        item['jobViewsCountOn'] = str(date.today())




        jobInfo = response.xpath('//table[@class="table table-hover table-no-border m-0"][1]')

        item['jobLink'] = response.url

        text = jobInfo.xpath('//tr/td[2]//text()').extract()

        itemText = response.xpath('//table[@class="table table-hover table-no-border m-0"][1]/tr/td[2]//text()').extract()

        if itemText is not None:

            str_list = [x.encode('ascii', 'ignore').decode('ascii').strip() for x in itemText if len(x.encode('ascii', 'ignore').decode('ascii').strip())&gt;1]




            label = response.xpath('//table[@class="table table-hover table-no-border m-0"][1]/tr/td[1]//text()').extract()

            str_list_label = [x.encode('ascii', 'ignore').decode('ascii').strip() for x in label if len(x.encode('ascii', 'ignore').decode('ascii').strip())&gt;1]




            numberOfVacancy = jobInfo.xpath('//tr[3]/td[2]/strong/text()').extract_first()

            item['numberOfVacancy'] = numberOfVacancy

            str_list_label = list(filter(lambda x : x != 'No. of Vacancy/s', str_list_label))

            str_list_label = list(filter(lambda x : x != 'Apply Before', str_list_label))




            for index, label in enumerate(str_list_label):

                if 'Category' in label:

                    item['jobCategory'] = str_list[index]

                if 'Job Level' in label:

                    item['jobLevel'] = str_list[index]

                if 'Employment Type' in label:

                    item['employementType'] = str_list[index]  

                if 'Salary' in label:

                    item['salary'] = str_list[index] 

                if 'Deadline' in label:

                    item['applicationDeadline'] = str_list[index][:20].strip()

                if 'Education Level' in label:

                    item['educationLevel'] = str_list[index]

                if 'Experience' in label:

                    item['experienceRequired'] = str_list[index]




        yield item

def parseJobLink(self, response):

item = MerojobItem()

item['employer'] = response.xpath('//span[@itemprop="name"]/text()').extract_first()

employerDesc = response.xpath('//div[@itemprop="description"]//text()').extract()

item['employerDesc'] = response.xpath('//div[@itemprop="description"]//text()').extract_first().encode('ascii', 'ignore').decode('ascii').strip()

employerDesc = response.xpath('//div[@itemprop="description"]//text()').extract()

item['employerDesc'] = "".join(x.encode('ascii', 'ignore').decode('ascii').strip() for x in employerDesc)

item['jobTitle'] = response.xpath('//h1[@itemprop="title"]/text()').extract_first()

item['jobViews'] = response.xpath('//span[@class="text-success"]/text()').extract_first()

item['jobViewsCountOn'] = str(date.today())

jobInfo = response.xpath('//table[@class="table table-hover table-no-border m-0"][1]')

item['jobLink'] = response.url

text = jobInfo.xpath('//tr/td[2]//text()').extract()

itemText = response.xpath('//table[@class="table table-hover table-no-border m-0"][1]/tr/td[2]//text()').extract()

if itemText is not None:

str_list = [x.encode('ascii', 'ignore').decode('ascii').strip() for x in itemText if len(x.encode('ascii', 'ignore').decode('ascii').strip())>1]

label = response.xpath('//table[@class="table table-hover table-no-border m-0"][1]/tr/td[1]//text()').extract()

str_list_label = [x.encode('ascii', 'ignore').decode('ascii').strip() for x in label if len(x.encode('ascii', 'ignore').decode('ascii').strip())>1]

numberOfVacancy = jobInfo.xpath('//tr[3]/td[2]/strong/text()').extract_first()

item['numberOfVacancy'] = numberOfVacancy

str_list_label = list(filter(lambda x : x != 'No. of Vacancy/s', str_list_label))

str_list_label = list(filter(lambda x : x != 'Apply Before', str_list_label))

for index, label in enumerate(str_list_label):

if 'Category' in label:

item['jobCategory'] = str_list[index]

if 'Job Level' in label:

item['jobLevel'] = str_list[index]

if 'Employment Type' in label:

item['employementType'] = str_list[index]

if 'Salary' in label:

item['salary'] = str_list[index]

if 'Deadline' in label:

item['applicationDeadline'] = str_list[index][:20].strip()

if 'Education Level' in label:

item['educationLevel'] = str_list[index]

if 'Experience' in label:

item['experienceRequired'] = str_list[index]

yield item

Introduction

GoodReads is a very good resource for info about books, authors and interesting quotations.

In this post, I will share a piece of code that will allow you to scrape for quotations from this site. The code is written for python’s Scrapy framework.

Getting Started

To get started with scraping quotes from your favorite author, first of all search for quotes by the author name in the quotes section.

Once you type the author’s name, you can look for css and xpath in the displayed results for finding pointers to scrape data.

Code For Spider

Now that we have data to scrape from, the next step is to create a spider that will scrape data from this page. A spider in scrapy is basically a class that you can use to scrape data from a location. You can find more info on scrapy here.

Basically, we want to loop over each “quoteDetail” section to get the author and quote text.


for sel in response.css('div.quoteDetails '):
quote = sel.css('div.quoteText::text').extract()
author = sel.css('div.quoteText a::text').extract_first()
item = GoodreadsItem()
item['author'] = author
item['quote'] = quote
yield item

Each quote gets extracted as a “GoodreadsItem” object.

Next, to scrape data from the next page, following code can be used:


checkNextPage = response.xpath('//a[@class="next_page"]').extract_first()
if(len(checkNextPage)>0):
nextPageLink = response.xpath('//a[@class="next_page"]/@href').extract_first()
nextPageFullUrl = response.urljoin(nextPageLink)
print(nextPageFullUrl)

Conclusion

That’s all the code needed for scraping. It’s quite easy and fun to scrape with Scrapy. Good luck!

pSovit

mixing literature, programming and some fun

Featured

Connecting to SQL Server using RODBC package in R

Uploading Unknown Format Excel File To A SQL Database

Post on ASP.NET Web API

A Knight’s Watch

7 Handy SQL Scripts for SQL Developers

Category / Scrapy

Scraping Jobs From MeroJob

Web Scraping Quotes From Good Reads

Scraping Data From Wikipedia

Web Scraping Data For Hollywood Actors/Actresses Info

Youtube Crawler