| regex | Hanna Perez [name] | The 年 year was 1999 | HaHa, @abc said that | GoGo wolverines #2026! |
|---|---|---|---|---|
| .{5} | Hanna | The 年 | HaHa, | GoGo |
| n{2} | nn | |||
| [0-9] | 1 | 2 | ||
| [0-9]+ | 1999 | 2026 | ||
| [a-zA-Z]+ | Hanna | The | HaHa | GoGo |
| \\s[a-zA-Z]+\\s | Perez | year | said | wolverines |
| \\s[[:alpha:]]+\\s | Perez | 年 | said | wolverines |
Week 7: Regular Expressions and Web Scraping
BIOSTAT 620: Introduction to Health Data Science
Dylan Cable, George G. Vega Yon, and Kelly Street
Today’s goals
Introduction to Regular Expressions
Understand the fundamentals of Web Scraping
Learn how to use an API
Regular Expressions: What is it?
A regular expression (shortened as regex or regexp; also referred to as rational expression) is a sequence of characters that define a search pattern. – Wikipedia
Regular Expressions: Why should you care?
We can use Regular Expressions for:
Validating data fields, email address, numbers, etc.
Searching text in various formats, e.g., addresses, there are many ways to write an address.
Replace text, e.g., different spellings,
Storm,Stôrm,StórmtoStorm.Remove text, e.g., tags from an HTML text,
<name>George</name>toGeorge.
Regex 101: Metacharacters
What makes regex special is metacharacters. While we can always use regex to match literals like dog, human, 1999, we only make use of all regex power when using metacharacters:
.Any character except new line^beginning of the text$end of the text[expression]Match any single character in “expression”, e.g.[0123456789]Any digit[0-9]Any digit in the range 0-9[a-z]Lower-case letters[A-Z]Upper-case letters[a-zA-Z]Lower or upper case letters.[a-zA-Z0-9]Any alpha-numeric
Regex 101: Metacharacters (cont. 1)
[^regex]Match any except those inregex, e.g.[^0123456789]Match any except a number[^0-9]Match anything except in the range 0-9[^./ ]any except dot, slash, and space.
Regex 101: Metacharacters (cont. 2)
Ranges, e.g., 0-9 or a-z, are locale- and implementation-dependent, meaning that the range of lower case letters may vary depending on the OS’s language. To solve for this problem, you could use Character classes. Some examples:
[[:lower:]]lower case letters in the current locale, could be[a-z][[:upper:]]upper case letters in the current locale, could be[A-Z][[:alpha:]]upper and lower case letters in the current locale, could be[a-zA-Z][[:digit:]]Digits: 0 1 2 3 4 5 6 7 8 9[[:alnum:]]Alpha numeric characters[[:alpha:]]and[[:digit:]].[[:punct:]]Punctuation characters: ! ” # $ % & ’ ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~.
For example, in the locale en_US, the word Ḧóla IS NOT fully matched by [a-zA-Z]+, but IT IS fully matched by [[:alpha:]]+.
Regex 101: Metacharacters (cont. 3)
Other important metacharacters:
\\swhite space, equivalent to[\\r\\n\\t\\f\\v ]|or (logical or).
Regex 101: Metacharacters (cont. 4)
These usually come together with specifying how many times (repetition):
regex?Zero or one match.regex*Zero or more matchesregex+One or more matchesregex{n,}At leastnmatchesregex{,m}at mostmmatchesregex{n,m}Betweennandmmatches.
Where regex is a regular expression
Regex 101: Metacharacters (cont. 5)
There are other operators that can be very useful,
(regex)Group capture.(?:regex)Group operation without capture.A(?=B)Look ahead (match). Find expression A where expression B follows.A(?!B)Look ahead (don’t match). Find expression A where expression B does not follow.(?<=B)ALook behind (match). Find expression A where expression B precedes.(?<!B)ALook behind (don’t match). Find expression A where expression B does not precede.
Group captures can be reused with \\1, \\2, …, \\n, referring to the first group, second group, etc.
More (great) information here https://regex101.com/
Regex 101: Examples
.{5}Match any character (except line end) five times.n{2}Match the letter n twice.[0-9]Match any number once.[0-9]+Match any number at least once.[a-zA-Z]+Match any lower or upper case letter at least once.\\s[a-zA-Z]+\\sMatch a space, any lower or upper case letter at least once, and a space.\\s[[:alpha:]]+\\sSame as before but this time with the[[:alpha:]]character class.
Regex 101: Examples (cont. 1)
Here we are extracting the first occurrence of the following regular expressions (using stringr::str_extract()):
Regex 101: Examples (cont. 3)
[a-zA-Z]+ [a-zA-Z]+Match two sets of letters separated by one space.[a-zA-Z]+\\s?Match one set of letters, maybe followed by a white space.([a-zA-Z]+)\\1Match any lower or upper case letter at least once and then match the same pattern again.(@|#)[a-z0-9]+Match either the@or#symbol, followed by one or more lower case letter or number.(?<=#|@)[a-z0-9]+Match one or more lower case letter or number that follows either the@or#symbol.\\[[a-z]+\\]Match the symbol[, at least one lower case letter, then symbol].
Regex 101: Examples (cont. 2)
| regex | Hanna Perez [name] | The 年 year was 1999 | HaHa, @abc said that | GoGo wolverines #2026! |
|---|---|---|---|---|
| [a-zA-Z]+ [a-zA-Z]+ | Hanna Perez | year was | abc said | GoGo wolverines |
| [a-zA-Z]+\\s? | Hanna | The | HaHa | GoGo |
| ([a-zA-Z]+)\\1 | nn | HaHa | GoGo | |
| (@|#)[a-z0-9]+ | @abc | #2026 | ||
| (?<=#|@)[a-z0-9]+ | abc | 2026 |
Regex 101: Examples (cont. 3)
Regex 101: Functions in R
Find in text:
grepl(),stringr::str_detect().Similar to
which(), which elements areTRUEgrep(),stringr::str_which()Replace the first instance:
sub(),stringr::str_replace()Replace all instances:
gsub(),stringr::str_replace_all()Extract text:
regmatches()plusregexpr(),stringr::str_extract()andstringr::str_extract_all().
Regex 101: Functions in R (cont. 1)
For example, like in Twitter, let’s create a regex that matches usernames or hashtags with the following pattern:
(@|#)([[:alnum:]]+)
| Code | @Hanna Perez [name] #html | The @年 year was 1999 | HaHa, @abc said that @z |
|---|---|---|---|
str_detect(text, pattern) or grepl(pattern, text) |
TRUE | TRUE | TRUE |
str_extract(text, pattern) or regmatches(...) |
@Hanna | @年 | @abc |
str_extract_all(text, pattern) |
@Hanna, #html | @年 | @abc, @z |
Regex 101: Functions in R (cont. 2)
Pattern: (@|#)([[:alnum:]]+)
| Code | @Hanna Perez [name] #html | The @年 year was 1999 | HaHa, @abc said that @z |
|---|---|---|---|
str_replace(text, pattern, "\\1justinbieber") or sub(...) |
@justinbieber Perez [name] #html | The @justinbieber year was 1999 | HaHa, @justinbieber said that @z |
str_replace_all(text, pattern, "\\1justinbieber") or gsub(...) |
@justinbieber Perez [name] #justinbieber | The @justinbieber year was 1999 | HaHa, @justinbieber said that @justinbieber |
Note: It renders oddly in the table, but there is no space or line break in the group replacement, \\1.
Data
This week we will continue using our text mining dataset:
# Where are we getting the data from
mts_url <- "https://github.com/dmcable/BIOSTAT620W26/raw/main/data/mtsamples/mtsamples.csv"
# Downloading the data to a tempfile (so it is destroyed afterwards)
tmp <- tempfile(pattern = "mtsamples", fileext = ".csv")
# We should be downloading this, ONLY IF this was not downloaded already.
# otherwise is just a waste of time.
if (!file.exists(tmp)) {
download.file(
url = mts_url,
destfile = tmp,
# method = "libcurl", timeout = 1000 (you may need this option)
)
}
# read the file
mtsamples <- read.csv(tmp, header = TRUE, row.names = 1)Regex Lookup Text: Tumor
For each entry, we want to know if it is tumor-related. For that we can use the following code:
# How many entries contain the word tumor?
sum(grepl("tumor", mtsamples$description, ignore.case = TRUE))[1] 80# Generating a column tagging tumor
mtsamples$tumor_related <- grepl("tumor", mtsamples$description, ignore.case = TRUE)
# Taking a look at a few examples
mtsamples$description[mtsamples$tumor_related == TRUE][1:3][1] " Transurethral resection of a medium bladder tumor (TURBT), left lateral wall."
[2] " Transurethral resection of the bladder tumor (TURBT), large."
[3] " Cystoscopy, transurethral resection of medium bladder tumor (4.0 cm in diameter), and direct bladder biopsy."Notice the ignore.case = TRUE. This is equivalent to transforming the text to lower case using tolower() before passing the text to the regular expression function.
Regex Lookup text: Pronoun of the patient
Now, let’s try to guess the pronoun of the patient. To do so, we could tag by using the words he, his, him, they, them, she, her:
What is the problem with this approach?
Regex Lookup text: Pronoun of the patient (cont. 1)
[1] "SUBJECTIVE:, This 23-year-old white female presents with complaint of allergies. She used to have allergies when she lived in Seattle but she thinks they are worse here. In the past, she has tried Claritin, and Zyrtec. Both worked for short time but then seemed to lose effectiveness. She has used Allegra also. She used that last summer and she began using it again two weeks ago. It does not appear to be working very well. She has used over-the-counter sprays but no prescription nasal sprays. She does have asthma but doest not require daily medication for this and does not think it is flaring up.,MEDICATIONS: , Her only medication currently is Ortho Tri-Cyclen and the Allegra.,ALLERGIES: , She has no known medicine allergies.,OBJECTIVE:,Vitals: Weight was 130 pounds and blood pressure 124/78.,HEENT: Her throat was mildly erythematous without exudate. Nasal mucosa was erythematous and swollen. Only clear drainage was seen. TMs were clear.,Neck: Supple without adenopathy.,Lungs: Clear.,ASSESSMENT:, Allergic rhinitis.,PLAN:,1. She will try Zyrtec instead of Allegra again. Another option will be to use loratadine. She does not think she has prescription coverage so that might be cheaper.,2. Samples of Nasonex two sprays in each nostril given for three weeks. A prescription was written as well."[1] "his"Regex Lookup text: Pronoun of the patient (cont. 1)
To correct this issue, we can make our regular expression more precise:
(?<=\W|^)(he|his|him|they|them|she|her)(?=\W|$)
Bit by bit this is:
(?<=regex)look behind for…\Wany non-alphanumeric character, this is equivalent to[^[:alnum:]],|or^the beginning of text
he|his|him...any of these words(?=regex)followed by…\Wany non-alphanumeric character, this is equivalent to[^[:alnum:]],|or$the end of the text.
Regex Lookup text: Pronoun of the patient (cont. 2)
Let’s use this new pattern:
Regex Lookup text: Pronoun of the patient (cont. 3)
Regex Extract Text: Type of Cancer
Imagine now that you need to see the types of cancer mentioned in the data.
For simplicity, let’s assume that, if specified, it is in the form of
TYPE cancer, i.e. single word.We are interested in the word before cancer, how can we capture this?
Regex Extract Text: Type of Cancer (cont 1.)
We can just try to extract the phrase "[some word] cancer", in particular, we could use the following regular expression:
[[:alnum:]-_]{4,}\s*cancer
Where
[[:alnum:]-_]{4,}captures any alphanumeric character, including-and_. Furthermore, for this match to work there must be at least 4 characters,\s*captures 0 or more white-spaces, andcancercaptures the word cancer
Regex Extract Text: Type of Cancer (cont. 2)
mtsamples$cancer_type <- str_extract(tolower(mtsamples$keywords), "[[:alnum:]-_]{4,}\\s*cancer")
table(mtsamples$cancer_type)
anal cancer bladder cancer breast cancer colon cancer
1 8 21 14
endometrial cancer esophageal cancer lung cancer ovarian cancer
5 2 13 1
papillary cancer prostate cancer uterine cancer
3 17 7 Web Scraping
Fundamentals of Web Scraping
What?
Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites – Wikipedia
How?
The
rvestR package provides various tools for reading and processing web data.Under the hood,
rvestis a wrapper of thexml2andhttrR packages.
(in the case of dynamic websites, take a look at selenium)
Web scraping raw HTML: Example
We would like to capture the table of COVID-19 death rates per country directly from Wikipedia.
library(rvest)
library(xml2)
# Reading the HTML table with the function xml2::read_html
covid <- read_html(
x = "https://en.wikipedia.org/wiki/COVID-19_pandemic_death_rates_by_country"
)
# Let's look at the the output
covid{html_document}
<html class="client-nojs vector-feature-language-in-header-enabled vector-feature-language-in-main-page-header-disabled vector-feature-page-tools-pinned-disabled vector-feature-toc-pinned-clientpref-1 vector-feature-main-menu-pinned-disabled vector-feature-limited-width-clientpref-1 vector-feature-limited-width-content-enabled vector-feature-custom-font-size-clientpref-1 vector-feature-appearance-pinned-clientpref-1 skin-theme-clientpref-day vector-sticky-header-enabled wp25eastereggs-enable-clientpref-1 vector-toc-available" lang="en" dir="ltr">
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
[2] <body class="skin--responsive skin-vector skin-vector-search-vue mediawik ...Web scraping raw HTML: Example (cont. 1)
We want to get the HTML table that shows up in the doc. To do so, we can use the function
xml2::xml_find_all()andrvest::html_table()The first will locate the place in the document that matches a given XPath expression.
XPath, XML Path Language, is a query language to select nodes in a XML document.
A nice tutorial can be found here
Modern Web browsers make it easy to use XPath!
Live Example! (inspect elements in Google Chrome, Mozilla Firefox, Internet Explorer, and Safari)
Web scraping with xml2 and the rvest package (cont. 2)
Now that we know the path, let’s use that and extract the table:
table <- xml_find_first(covid, xpath = '//div[@class="sticky-table-scroll"]/table')
#table <- xml_find_all(covid, xpath = '/html/body/div[3]/div/div[3]/main/div[3]/div[3]/div[1]/div[6]/table')
print(table) # collection of tables in XML{html_node}
<table class="wikitable sortable sort-under static-row-numbers mw-datatable plainrowheaders sticky-table-row1 sticky-table-col1" style="text-align: right;">
[1] <caption>\n<a href="/wiki/COVID-19_pandemic" title="COVID-19 pandemic">CO ...
[2] <tbody>\n<tr>\n<th scope="col">Country\n</th>\n<th scope="col">\n<abbr ti ...Web scraping with xml2 and the rvest package (cont. 3)
We’re getting closer!
# A tibble: 240 × 4
Country `Deaths / million` Deaths Cases
<chr> <chr> <chr> <chr>
1 World[a] 893 7,110,148 779,026,342
2 Peru 6,603 221,067 4,533,436
3 Bulgaria 5,680 38,777 1,341,016
4 North Macedonia 5,429 9,991 352,093
5 Bosnia and Herzegovina 5,119 16,407 404,289
6 Hungary 5,072 49,124 2,240,565
7 Croatia 4,813 18,805 1,371,301
8 Slovenia 4,686 9,914 1,366,726
9 Georgia 4,519 17,151 1,864,337
10 Montenegro 4,317 2,654 251,280
# ℹ 230 more rowsWeb scraping with xml2 and the rvest package (cont. 4)
There it is!
[1] "World[a]"
[2] "Peru"
[3] "Bulgaria"
[4] "North Macedonia"
[5] "Bosnia and Herzegovina"
[6] "Hungary"
[7] "Croatia"
[8] "Slovenia"
[9] "Georgia"
[10] "Montenegro"
[11] "Czech Republic"
[12] "Latvia"
[13] "Moldova"
[14] "Slovakia"
[15] "Greece"
[16] "San Marino"
[17] "United States"
[18] "Romania"
[19] "Lithuania"
[20] "United Kingdom"
[21] "Brazil"
[22] "Italy"
[23] "Chile"
[24] "Martinique"
[25] "Poland"
[26] "Armenia"
[27] "Gibraltar"
[28] "Belgium"
[29] "Paraguay"
[30] "Trinidad and Tobago"
[31] "Argentina"
[32] "Portugal"
[33] "European Union[b]"
[34] "Sweden"
[35] "Russia"
[36] "Colombia"
[37] "Aruba"
[38] "Ukraine"
[39] "Serbia"
[40] "Guadeloupe"
[41] "France"
[42] "Mexico"
[43] "Spain"
[44] "Bermuda"
[45] "Guam"
[46] "Estonia"
[47] "Austria"
[48] "Tunisia"
[49] "French Polynesia"
[50] "Saint Lucia"
[51] "Uruguay"
[52] "Liechtenstein"
[53] "Suriname"
[54] "Sint Maarten"
[55] "Bahamas"
[56] "Malta"
[57] "Barbados"
[58] "Germany"
[59] "Finland"
[60] "Grenada"
[61] "Ecuador"
[62] "Panama"
[63] "Andorra"
[64] "Republic of Ireland"
[65] "Lebanon"
[66] "Kosovo"
[67] "Bolivia"
[68] "Costa Rica"
[69] "Puerto Rico"
[70] "Hong Kong"
[71] "Montserrat"
[72] "Monaco"
[73] "Belize"
[74] "Denmark"
[75] "British Virgin Islands"
[76] "Curaçao"
[77] "South Africa"
[78] "Iran"
[79] "Switzerland"
[80] "Guyana"
[81] "Collectivity of Saint Martin"
[82] "Antigua and Barbuda"
[83] "Jersey"
[84] "Luxembourg"
[85] "United States Virgin Islands"
[86] "Cyprus"
[87] "Caribbean Netherlands"
[88] "Canada"
[89] "Namibia"
[90] "Israel"
[91] "French Guiana"
[92] "Isle of Man"
[93] "Seychelles"
[94] "Netherlands"
[95] "Jamaica"
[96] "Albania"
[97] "Jordan"
[98] "Saint Vincent and the Grenadines"
[99] "Eswatini"
[100] "Turkey"
[101] "Botswana"
[102] "Guatemala"
[103] "Dominica"
[104] "New Caledonia"
[105] "Malaysia"
[106] "Palestine"
[107] "Honduras"
[108] "Réunion"
[109] "Guernsey"
[110] "Norway"
[111] "Azerbaijan"
[112] "Bahrain"
[113] "Saint Kitts and Nevis"
[114] "Oman"
[115] "Australia"
[116] "Fiji"
[117] "Kazakhstan"
[118] "Turks and Caicos Islands"
[119] "Libya"
[120] "Northern Mariana Islands"
[121] "New Zealand"
[122] "Anguilla"
[123] "Mauritius"
[124] "Cabo Verde"
[125] "Wallis and Futuna"
[126] "Belarus"
[127] "Cuba"
[128] "Sri Lanka"
[129] "Taiwan"
[130] "American Samoa"
[131] "South Korea"
[132] "El Salvador"
[133] "Mongolia"
[134] "Mayotte"
[135] "Maldives"
[136] "Japan"
[137] "Philippines"
[138] "Indonesia"
[139] "Federated States of Micronesia"
[140] "Iraq"
[141] "Palau"
[142] "Kuwait"
[143] "Faroe Islands"
[144] "Cayman Islands"
[145] "Iceland"
[146] "Thailand"
[147] "Saint Barthélemy"
[148] "Morocco"
[149] "Vietnam"
[150] "Marshall Islands"
[151] "Nepal"
[152] "Brunei"
[153] "Dominican Republic"
[154] "Greenland"
[155] "India"
[156] "Myanmar"
[157] "Singapore"
[158] "Zimbabwe"
[159] "Sao Tome and Principe"
[160] "Saint Pierre and Miquelon"
[161] "Lesotho"
[162] "Saudi Arabia"
[163] "Solomon Islands"
[164] "Qatar"
[165] "United Arab Emirates"
[166] "Egypt"
[167] "Venezuela"
[168] "Mauritania"
[169] "Zambia"
[170] "Afghanistan"
[171] "Comoros"
[172] "Kiribati"
[173] "Cambodia"
[174] "Bangladesh"
[175] "Macau"
[176] "Djibouti"
[177] "Algeria"
[178] "Kyrgyzstan"
[179] "Samoa"
[180] "Gambia"
[181] "Syria"
[182] "Cook Islands"
[183] "Malawi"
[184] "Gabon"
[185] "Pakistan"
[186] "Tonga"
[187] "Senegal"
[188] "Rwanda"
[189] "Kenya"
[190] "Sudan"
[191] "Equatorial Guinea"
[192] "Timor-Leste"
[193] "Tuvalu"
[194] "Laos"
[195] "China[c]"
[196] "Nauru"
[197] "Guinea-Bissau"
[198] "Uganda"
[199] "Somalia"
[200] "Haiti"
[201] "Cameroon"
[202] "Mozambique"
[203] "Papua New Guinea"
[204] "Republic of the Congo"
[205] "Ethiopia"
[206] "Yemen"
[207] "Liberia"
[208] "Angola"
[209] "Madagascar"
[210] "Vanuatu"
[211] "Ghana"
[212] "Nicaragua"
[213] "Guinea"
[214] "Mali"
[215] "Togo"
[216] "Eritrea"
[217] "Uzbekistan"
[218] "Ivory Coast"
[219] "Bhutan"
[220] "Central African Republic"
[221] "Burkina Faso"
[222] "Sierra Leone"
[223] "Democratic Republic of the Congo"
[224] "Nigeria"
[225] "South Sudan"
[226] "Tanzania"
[227] "Niger"
[228] "Tajikistan"
[229] "Benin"
[230] "Chad"
[231] "Burundi"
[232] "Vatican City"
[233] "Niue"
[234] "Falkland Islands"
[235] "Saint Helena, Ascension and Tristan da Cunha"
[236] "Pitcairn Islands"
[237] "Tokelau"
[238] "North Korea"
[239] "Turkmenistan"
[240] ".mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}body.skin-vector-2022 .mw-parser-output .reflist-columns-2{column-width:27em}body.skin-vector-2022 .mw-parser-output .reflist-columns-3{column-width:22.5em}.mw-parser-output .references[data-mw-group=upper-alpha]{list-style-type:upper-alpha}.mw-parser-output .references[data-mw-group=upper-roman]{list-style-type:upper-roman}.mw-parser-output .references[data-mw-group=lower-alpha]{list-style-type:lower-alpha}.mw-parser-output .references[data-mw-group=lower-greek]{list-style-type:lower-greek}.mw-parser-output .references[data-mw-group=lower-roman]{list-style-type:lower-roman}.mw-parser-output div.reflist-liststyle-upper-alpha .references{list-style-type:upper-alpha}.mw-parser-output div.reflist-liststyle-upper-roman .references{list-style-type:upper-roman}.mw-parser-output div.reflist-liststyle-lower-alpha .references{list-style-type:lower-alpha}.mw-parser-output div.reflist-liststyle-lower-greek .references{list-style-type:lower-greek}.mw-parser-output div.reflist-liststyle-lower-roman .references{list-style-type:lower-roman}\n^ Countries which do not report data for a column are not included in that column's world total.\n\n^ Data on member states of the European Union are individually listed, but are also summed here for convenience. They are not double-counted in world totals.\n\n^ Does not include special administrative regions (Hong Kong and Macau) or Taiwan."* Web APIs
What?
A Web API is an application programming interface for either a web server or a web browser. – Wikipedia
Some examples include: twitter API, facebook API, Gene Ontology API
How?
You can request data, the GET method, post data, the POST method, and do many other things using the HTTP protocol.
How in R?
For this part, we will be using the httr() package, which is a wrapper of the curl() package, which in turn provides access to the curl library that is used to communicate with APIs.
* Web APIs with curl
Structure of a URL (source: “HTTP: The Protocol Every Web Developer Must Know - Part 1”)
* Web APIs with curl
Under the hood, the httr (and thus curl) sends request somewhat like this
A get request (-X GET) to https://google.com, which also includes (-w) the following: content_type and http_code:
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
text/html; charset=UTF-8
301We use the httr R package to make life easier.
* Web API Example 1: Gene Ontology
We will make use of the Gene Ontology API.
We want to know what genes (human or not) are involved in the function antiviral innate immune response (go term GO:0140374), looking only at those annotations that have evidence code ECO:0000006 (experimental evidence):
library(httr)
go_query <- GET(
url = "http://api.geneontology.org/",
path = "api/bioentity/function/GO:0140374/genes",
query = list(
evidence = "ECO:0000006",
relationship_type = "involved_in"
),
# May need to pass this option to curl to allow to wait for at least
# 60 seconds before returning error.
config = config(
connecttimeout = 60
)
)We could have also passed the full URL directly…
* Web API Example 1: Gene Ontology (cont. 1)
Let’s take a look at the curl call:
curl -X GET "http://api.geneontology.org/api/bioentity/function/GO:0140374/genes?evidence=ECO%3A0000006&relationship_type=involved_in" -H "accept: application/json"What httr::GET() does:
> go_query$request
## <request>
## GET http://api.geneontology.org/api/bioentity/function/GO:0140374/genes?evidence=ECO%3A0000006&relationship_type=involved_in
## Output: write_memory
## Options:
## * useragent: libcurl/7.58.0 r-curl/4.3 httr/1.4.1
## * connecttimeout: 60
## * httpget: TRUE
## Headers:
## * Accept: application/json, text/xml, application/xml, */** Web API Example 1: Gene Ontology (cont. 2)
Let’s take a look at the response:
Response [https://api.geneontology.org/api/bioentity/function/GO:0140374/genes?evidence=ECO%3A0000006&relationship_type=involved_in]
Date: 2026-02-26 01:41
Status: 200
Content-Type: application/json
Size: 119 kBRemember the codes:
- 1xx: Information message
- 2xx: Success
- 3xx: Redirection
- 4xx: Client error
- 5xx: Server error
* Web API Example 1: Gene Ontology (cont. 3)
We can extract the results using the httr::content() function:
dat <- content(go_query)
dat <- lapply(dat$associations, function(a) {
data.frame(
Gene = a$subject$id,
taxon_id = a$subject$taxon$id,
taxon_label = a$subject$taxon$label
)
})
dat <- do.call(rbind, dat)
str(dat)'data.frame': 100 obs. of 3 variables:
$ Gene : chr "UniProtKB:A0A8D0WX68" "UniProtKB:A0A5F8HH92" "UniProtKB:A0A5F8HG58" "UniProtKB:A0A9L0S3I7" ...
$ taxon_id : chr "NCBITaxon:9823" "NCBITaxon:13616" "NCBITaxon:13616" "NCBITaxon:9796" ...
$ taxon_label: chr "Sus scrofa" "Monodelphis domestica" "Monodelphis domestica" "Equus caballus" ...* Web API Example 1: Gene Ontology (cont. 4)
The structure of the result will depend on the API. In this case, the output was a JSON file, so the content function returns a list in R. In other scenarios it could return an XML object.
knitr::kable(head(dat),
caption = "Genes experimentally annotated with the function\
**antiviral innate immune response** (GO:0140374)"
)| Gene | taxon_id | taxon_label |
|---|---|---|
| UniProtKB:A0A8D0WX68 | NCBITaxon:9823 | Sus scrofa |
| UniProtKB:A0A5F8HH92 | NCBITaxon:13616 | Monodelphis domestica |
| UniProtKB:A0A5F8HG58 | NCBITaxon:13616 | Monodelphis domestica |
| UniProtKB:A0A9L0S3I7 | NCBITaxon:9796 | Equus caballus |
| UniProtKB:A0A5F8GEE1 | NCBITaxon:13616 | Monodelphis domestica |
| UniProtKB:Q5U533 | NCBITaxon:8355 | Xenopus laevis |
* Web API Example 2: Using Tokens
Sometimes, APIs are not completely open, you need to register.
The API may require to login (user+password), or pass a token.
In this example, I’m using a token which I obtained here
You can find information about the National Centers for Environmental Information API here
* Web API Example 2: Using Tokens (cont. 1)
The way to pass the token will depend on the API service.
Some require authentication, others need you to pass it as an argument of the query, i.e., directly in the URL.
In this case, we pass it on the header.
This is equivalent to using the following query
Note: This won’t run, you need to get your own token
* Web API Example 2: Using Tokens (cont. 2)
Again, we can recover the data using the content() function:
* Web API Example 3: HHS health recommendation
Here is a last example. We will use the Department of Health and Human Services API for “[…] demographic-specific health recommendations” (details at health.gov)
health_advises <- GET(
url = "https://odphp.health.gov",
path = "myhealthfinder/api/v4/myhealthfinder.json",
query = list(
lang = "en",
age = "32",
sex = "male",
tobaccoUse = "no" # Note: v4 often uses "yes"/"no" instead of 0/1
),
config = c(
add_headers(Accept = "application/json"),
timeout(60)
)
)* Web API Example 3: HHS health recommendation (cont. 1)
Let’s see the response
Response [https://odphp.health.gov/myhealthfinder/api/v4/myhealthfinder.json?lang=en&age=32&sex=male&tobaccoUse=no]
Date: 2026-02-26 01:41
Status: 200
Content-Type: application/json
Size: 236 kB
{
"Result": {
"Error": "False",
"Total": 16,
"Query": {
"ApiVersion": "4",
"ApiType": "myhealthfinder",
"TopicId": null,
"CategoryId": null,
"Keyword": null,
...* Web API Example 3: HHS health recommendation (cont. 2)
# Extracting the content
health_advises_ans <- content(health_advises)
# Getting the titles
txt <- with(health_advises_ans$Result$Resources, c(
sapply(All$Resource, "[[", "Title")
))
cat(txt, sep = "; ")Get Your Blood Pressure Checked; Hepatitis C Screening: Questions for the Doctor; Get Tested for HIV; Get Vaccines to Protect Your Health (Adults Ages 19 to 49 Years); Testing for Syphilis: Questions for the Doctor; Drink Alcohol Only in Moderation; Talk with Your Doctor About Anxiety; Talk with Your Doctor About Depression; Talk with Your Doctor About Drug Misuse and Substance Use Disorder; Eat Healthy; Protect Yourself from Hepatitis B; Testing for Latent Tuberculosis: Questions for the Doctor
Summary
We learned about regular expressions with the package stringr (a wrapper of stringi)
We can use regular expressions to detect (
str_detect()), replace (str_replace()), and extract (str_extract()) expressions.We looked at web scraping using the rvest package (a wrapper of xml2).
We extracted elements from the HTML/XML using
xml_find_all()with XPath expressions.We also used the
html_table()function from rvest to extract tables from HTML documents.We took a quick review on Web APIs and the Hyper-text-transfer-protocol (HTTP).
We used the httr R package (wrapper of curl) to make
GETrequests to various APIsWe even showed an example using a token passed via the
header.Once we got the responses, we used the
content()function to extract the message of the response.
Detour on CURL options
Sometimes you will need to change the default set of options in CURL. You can checkout the list of options in curl::curl_options(). A common hack is to extend the time-limit before dropping the conection, e.g.:
Using the Health IT API from the US government, we can obtain the Electronic Prescribing Adoption and Use by County (see docs here)
The problem is that it usually takes longer to get the data, so we pass the config option connecttimeout (which corresponds to the flag --connect-timeout) in the curl call (see next slide)
Detour on CURL options (cont. 1)
> ans$request
# <request>
# GET https://dashboard.healthit.gov/api/open-api.php?source=AHA_2008-2015.csv®ion=Michigan&period=2015
# Output: write_memory
# Options:
# * useragent: libcurl/7.58.0 r-curl/4.3 httr/1.4.1
# * connecttimeout: 60
# * httpget: TRUE
# Headers:
# * Accept: application/json, text/xml, application/xml, */*Regular Expressions: Email validation
This is the official regex for email validation implemented by RCF 5322
(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08
\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?
:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[
0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0
-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\
x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])See the corresponding post in StackOverflow