Retrieving results

Collecting more than 10 results

⚠️ The number of results you will be able to return will be dependent on your level of access:

• Example: 10 pages
• Limited: 100 pages
• Top: 10,000 pages

In the introduction we showcased how to do a quick search. But it is very likely that you will need to collect more than 10 results. How would this task be performed?

Firstly, load purrr and dplyr:

suppressPackageStartupMessages(library(dplyr))
library(purrr)
library(timetravelData)

Once you have the packages loaded, next set up the iterator by firstly getting the number of hits in the database:

query <- "crocodile shoot +hunting -shoes"
total_hits <- query_hits(query)
iterator <- seq(0, total_hits, by = 10)

• Total hits: 10128
• Iterator length (number of requests to perform): 1013

You can now iterate over the search in the database. In this example I only show 10 as in the case of over query, we have to query 700 request to bring back ALL the information:

out <- iterator[1:5] %>% map_dfr(., ~query_text(query, from_pos = .x))

Finally, we can use the same itterator idea to collect the meta information for all the pages:

unique(out$guid_hash)[1:5] %>% map_dfr(document_meta)