Scraping RU Sociology department

In this last part of scraping names, we will show how to mimic a human webbrowsing activity. We will use RSelenium / Selenium for this. This can be a bit complicated to get up and running. For example, you need to hava Java installed.

Especially the R documentation is not great. In R, there is a relatively new package selendir which looks promising.

Because we will be interacting with websites, we did not show our output on this page.

1 Preparation

1.1 clean up

rm(list = ls())
gc()

1.2 general custom R functions

• fpackage.check: Check if packages are installed (and install if not) in R
• fsave: save data with time stamp in correct directory
• fload: load R-objects under new names
• fshowdf: Print objects (tibble / data.frame) nicely on screen in .Rmd.

fpackage.check <- function(packages) {
    lapply(packages, FUN = function(x) {
        if (!require(x, character.only = TRUE)) {
            install.packages(x, dependencies = TRUE)
            library(x, character.only = TRUE)
        }
    })
}

fsave <- function(x, file, location = "./local/", ...) {
    if (!dir.exists(location))
        dir.create(location)
    datename <- substr(gsub("[:-]", "", Sys.time()), 1, 8)
    totalname <- paste(location, datename, file, sep = "")
    print(paste("SAVED: ", totalname, sep = ""))
    save(x, file = totalname)
}

fload <- function(fileName) {
    load(fileName)
    get(ls()[ls() != "fileName"])
}

fshowdf <- function(x, digits = 2, ...) {
    knitr::kable(x, digits = digits, "html", ...) %>%
        kableExtra::kable_styling(bootstrap_options = c("striped", "hover")) %>%
        kableExtra::scroll_box(width = "100%", height = "300px")
}

1.3 necessary packages

1.3.1 R

packages = c("RSelenium", "rvest", "tidyverse", "netstat", "pingr", "stringr")
fpackage.check(packages)
rm(packages)

1.3.2 Python

Install the newest version of the selenium module inside a virtual environment, by typing in your terminal the below command:

pip install -U selenium

and import necessary modules

import selenium
import time

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

1.4 set up Selenium

1.4.1 R

# find a free port
port <- netstat::free_port(random = TRUE)

# ping port for confirmation
pingr::ping_port("www.jochemtolsma.nl", port = port)
ping_port("r-project.org")

Set up selenium server and browser (firefox is recommended) An issue with RSelenium::rsDriver is that it is configured to launch the Selenium server with the latest versions of Chrome and Firefox drivers, even if you have specified the other browser to run. To solve this, pass in NULL in the Chrome driver version chromever!

rD <- rsDriver(browser = "firefox", port = port, chromever = NULL, verbose = FALSE)
remDr <- rD[["client"]]

If at any time you wish to terminate the process, use the code:

pid <- rD$server$process$get_pid()  #get process id
system(paste0("Taskkill /F /T", " /PID ", pid))

1.4.2 Python

remDr = webdriver.Firefox()

If at any time you wish to terminate the process, use the code:

remDr.close()

---

2 Scraping scholar names

2.1 R

# navigate to RU staff page
remDr$navigate("https://www.ru.nl/en/search/scope/staff/")

# handle cookies
remDr$findElement(using = "css", value = ".agree-button")$highlightElement()
remDr$findElement(using = "css", value = ".agree-button")$clickElement()

cookies <- remDr$getAllCookies()
saveRDS(cookies, "cookies.rds")
remDr$addCookie(name = cookies[[1]][["name"]], value = cookies[[1]][["value"]])

remDr$maxWindowSize()

We want to get scholars from the sociology department; so we use the website’s collapsible lists and expand content under Radboud University.

We use the highlightElement() function to demonstrate where we are on the website.

remDr$findElement(using = "class name", value = "facet-list__expand-sub")$highlightElement()
remDr$findElement(using = "class name", value = "facet-list__expand-sub")$clickElement()

# we have to expand the list
remDr$findElement(using = "class name", value = "btn--show-hide")$highlightElement()
remDr$findElement(using = "class name", value = "btn--show-hide")$clickElement()

Expand content under Faculty of social science (dep id 961) using its xpath.
Note, this is the html:

<button class="facet-list__expand-sub is-active" aria-controls="toggle-staff-department-961" aria-expanded="false" data-once="ruHierarchicalFacetsClick" style=""><span class="visually-hidden">Hide filters</span>-</button>

This translates into the xpath shown below. Note here we use a compound class!
Difficulties with xpath? Have a look here

There seems to be a small bug in the website which breaks our code. Let us try to fix this in class.

remDr$findElement(using = "xpath", value = "//button[@class='facet-list__expand-sub' and @aria-controls='toggle-staff-department-961']")$highlightElement()

# If you would follow the complete html structure you will arrive at the same point.
remDr$findElement(using = "xpath", value = "/html/body/div[1]/div/div[1]/aside/div/div[27]/div/ul/li/div/ul/li[9]/button")$highlightElement()

# and click
remDr$findElement(using = "xpath", value = "//button[@class='facet-list__expand-sub' and @aria-controls='toggle-staff-department-961']")$clickElement()

Here we encounter the bug: not all departments are shown. You have to click on less and more.

Assignment: Let your crawler click on less and more (of fix the bug in a better way)

Once done, we can continue

#click on sociology
remDr$findElement(using = "xpath", value = "//input[@type='checkbox' and @class='facets-checkbox' and @id='staff-department-992']")$clickElement()

#among all staff members, select researchers

remDr$findElement(using = "xpath", value = "//input[@type='checkbox' and @class='facets-checkbox' and @id='staff-staff-researchers']")$highlightElement()

#more complicated but insightful 
remDr$findElement(using = "id", value = "block-staffstaff")$highlightElement()

remDr$findElement(using = "id", value = "block-staffstaff")$getElementText()

#first get categories:
remDr$findElement(using = "id", value = "block-staffstaff")$getElementText() %>% #get element text;
  unlist(.) %>% #unlist
  strsplit(., "\n") %>% #split by \n
  .[[1]] %>% #take the first element from the list
  .[-1] -> staff_cats #remove the header "staff"

#take the indicator of researchers
researchers <- grep("Researchers", staff_cats)   

#check the corresponding checkbox
remDr$findElements(using = "class name", value = "facets-checkbox")[[researchers]]$clickElement()

Naturally, this can also be achieved by using the URL https://www.ru.nl/en/search/scope/staff/staff-department/X/staff-staff/researchers, where X should be replaced by department-id, which can be stored in a list beforehand (this is useful if we want multiple departments of a faculty)

Now we have all sociology researchers (or of any other department) but these are spread across multiple pages.
To know how many pages there are we navigate to the “last” page first, save the last page, go back and loop through all pages:

remDr$findElements("class name", "pager__link-text")[[2]]$clickElement()

#get the page number of this page
npages <- remDr$findElement(using = "xpath", "//a[@aria-current='page']")$getElementText() %>%
  unlist(.) %>% #unlist
  strsplit(., "\n") %>% #split by \n
  .[[1]] %>%
  .[2] %>%
  as.numeric(.)

#go back to the first page
remDr$findElements("class name", "pager__link-text")[[1]]$clickElement()

#get all elements with class name "card__title"; these refer to names of researchers, on the first page. Retrieve the text:
names_page <- unlist(lapply(remDr$findElements("class name", "card__title"),
                     function(x) x$getElementText()))

#now repeat this for the other pages in 1:npages-1
page1 <- remDr$getCurrentUrl()

for ( i in 1:(npages-1)) { 
  #increment page 
  newpage <- sub("page=0$", paste0("page=", i), page1)
  #navigate to the page
  remDr$navigate(newpage)
  #retrieve names
  new_names <- unlist(lapply(remDr$findElements("class name", "card__title"),
                     function(x) x$getElementText()))
  #and add to the names_page object:
  names_page <- c(names_page, new_names)
  #add sleeper:
  Sys.sleep(2)
}

#all names!
print(names_page)
length(names_page)

2.2 Python

#navigate to RU staff page
remDr.get("https://www.ru.nl/en/search/scope/staff/")

remDr.maximize_window()

#we want to get scholars from the sociology department; so we use the website's collapsible lists
#expand content under Radboud University
remDr.find_element(By.CLASS_NAME, "facet-list__expand-sub").click()

#expand content under Faculty of social science (dep id 961)
#using its xpath
remDr.find_element(By.XPATH, "//button[@class='facet-list__expand-sub' and @aria-controls='toggle-staff-department-961']").click()

#click on sociology
remDr.find_element(By.XPATH, "//input[@type='checkbox' and @class='facets-checkbox' and @id='staff-department-992']").click()

#among all staff members, select researchers
#get the indicator of the checkbox that is "Researchers";

#first get categories:
staff_cats = remDr.find_element(By.ID, "block-staffstaff").text.split("\n")[1:]

#take the indicator of "researchers"
researchers = [i for i, cat in enumerate(staff_cats) if "Researchers" in cat][0]

#check the corresponding checkbox
remDr.find_elements(By.CLASS_NAME, "facets-checkbox")[researchers].click()

#naturally this can also be achieved by using the URL https://www.ru.nl/en/search/scope/staff/staff-department/X/staff-staff/researchers, where X should be replaced by department-id, which can be stored in a list beforehand (this is useful if we want multiple departments of a faculty)

##################################################################################################

#now we have all sociology researchers (or of any other department);
#but this is spread across multiple pages;
#to know how many pages there are - and thus through how many we must scrape - we navigate to the "last" page first;
remDr.find_elements(By.CLASS_NAME, "pager__link-text")[1].click()

#get the page number of this page
npages = int(remDr.find_element(By.XPATH, "//a[@aria-current='page']").text.split("\n")[1])
  
#go back to the first page
remDr.find_elements(By.CLASS_NAME, "pager__link-text")[0].click()

#get all elements with class name "card__title"; these refer to names of researchers, on the first page. Retrieve the text:
names_page = [elem.text for elem in remDr.find_elements(By.CLASS_NAME, "card__title")]

#now repeat this for the other pages
page1 = remDr.current_url

for i in range(1, npages):
  #increment page
  newpage = page1.replace("page=0", f"page={i}")
  print(str("navigating to: ") + newpage)
  time.sleep(0.5)
  #navigate to the page
  remDr.get(newpage)
  time.sleep(0.5)
  #retrieve names
  new_names = [elem.text for elem in remDr.find_elements(By.CLASS_NAME, "card__title")]
  #and add to the names_page object:
  names_page.extend(new_names)
  #add sleeper:
  time.sleep(2)

#all names!
print(names_page)
len(names_page)

---

Copyright © 2024 Jochem Tolsma