Some of the APIs require authentication:
cou | provider | label | access |
---|---|---|---|
USA | BEA | Bureau of Economic Analysis | register |
NLD | CBS | Statistics Netherlands | free |
DEU | GENESIS | Destatis | paid |
GBR | ONS | Office for National Statistics | register |
NOR | SSB | Statistics Norway | free |
AUT | STATAT | Statistics Austria | free |
CAN | WDS | Statistics Canada | IP |
Create an R script containing a list with your keys or just modify the the example script and save it with gpg encryption at inst/apiKey.R.gpg
.
apiKey <- list(
BEA = list(
userid = "29UG8ZKT-JN9R5N2Q")
,
GENESIS = list(
kennung = "8XXHKY78",
passwort = "3QYYTQ89")
,
ONS = list(
apikey = "k647uahw")
)
To prevent compromising the API keys, it is suggested to encrypt the file. If you are on Windows, you will need to install GnuPG first. The nsoApiGPG function allows to specify a directory to an executable using the gpg
parameter.
file.apiKey.enc <- system.file("apiKey.R.gpg", package = "nsoApi")
nsoApi::nsoApiGPG(file = file.apiKey.enc,
gpg = NULL,
passphrase = Sys.getenv("NSOAPIGPG"),
keep = FALSE
)
## [1] TRUE
api.param <- list(
DATASETNAME = "GDPbyIndustry",
METHOD = "GETPARAMETERVALUES",
RESULTFORMAT = "JSON",
USERID = apiKey$BEA$userid,
PARAMETERNAME = "TABLEID"
)
## req.uri <- beaAPI(api.param = api.param, curl = curl, query = TRUE)
## tt <- RCurl::getURL(req.uri, curl = curl)
tablelist <- beaAPI(api.param = api.param, curl = curl)
tablelist$BEAAPI$Request$RequestParam
## ParameterName ParameterValue
## 1 RESULTFORMAT JSON
## 2 PARAMETERNAME TABLEID
## 3 DATASETNAME GDPBYINDUSTRY
## 4 METHOD GETPARAMETERVALUES
## 5 USERID 7023E825-15FF-488D-B8D9-D70E6F67D439
Show first 5 tables
Key | Desc |
---|---|
1 | Value Added by Industry (A) (Q) |
5 | Value Added by Industry as a Percentage of Gross Domestic Product (A) (Q) |
6 | Components of Value Added by Industry (A) |
7 | Components of Value Added by Industry as a Percentage of Value Added (A) |
8 | Chain-Type Quantity Indexes for Value Added by Industry (A) (Q) |
api.param <- list(
DATASETNAME = "GDPBYINDUSTRY",
FREQUENCY = "A",
INDUSTRY = "11",
METHOD = "GETDATA",
RESULTFORMAT = "JSON",
USERID = apiKey$BEA$userid,
TABLEID = "1",
YEAR = paste(as.character(c(2008:2015)), collapse = ",")
)
datalist <- beaAPI(api.param = api.param, curl = curl)
knitr::kable(datalist$BEAAPI$Results$Data)
TableID | Frequency | Year | Quarter | Industry | IndustrYDescription | DataValue | NoteRef |
---|---|---|---|---|---|---|---|
1 | A | 2008 | 2008 | 11 | Agriculture, forestry, fishing, and hunting | 154.5 | 1 |
1 | A | 2009 | 2009 | 11 | Agriculture, forestry, fishing, and hunting | 137.7 | 1 |
1 | A | 2010 | 2010 | 11 | Agriculture, forestry, fishing, and hunting | 160.2 | 1 |
1 | A | 2011 | 2011 | 11 | Agriculture, forestry, fishing, and hunting | 197.2 | 1 |
1 | A | 2012 | 2012 | 11 | Agriculture, forestry, fishing, and hunting | 185.8 | 1 |
1 | A | 2013 | 2013 | 11 | Agriculture, forestry, fishing, and hunting | 221.8 | 1 |
1 | A | 2014 | 2014 | 11 | Agriculture, forestry, fishing, and hunting | 203.2 | 1 |
1 | A | 2015 | 2015 | 11 | Agriculture, forestry, fishing, and hunting | 175.2 | 1 |
First, we define the API URI and the dataset ID
api <- "https://opendata.cbs.nl/ODataApi/OData/"
DSD <- "82572ENG" # Input-Output: "83068ENG"
Next, we return the members of a dimension
scheme <- "SectorBranchesSIC2008"
## print query
cbsODataAPI(api=api, DSD=DSD, scheme=scheme, query=TRUE)
## [1] "https://opendata.cbs.nl/ODataApi/OData/82572ENG/SectorBranchesSIC2008"
cbs_dimension <- cbsODataAPI(api=api, DSD=DSD, scheme=scheme, query=FALSE)
knitr::kable(cbs_dimension[1:n_table, 1:2])
Key | Title |
---|---|
300025 | A-U All economic activities |
301000 | A Agriculture, forestry and fishing |
301100 | 01 Agriculture |
304300 | 02 Forestry and logging |
305000 | 03 Fishing and aquaculture |
Finally, we return a dataset
scheme <- "TypedDataSet"
# print query
cbsODataAPI(api=api, DSD=DSD, scheme=scheme, query=TRUE)
## [1] "https://opendata.cbs.nl/ODataApi/OData/82572ENG/TypedDataSet"
cbs_data <- cbsODataAPI(api=api, DSD=DSD, scheme=scheme, query=FALSE)
knitr::kable(cbs_data[1:n_table, 1:6])
ID | SectorBranchesSIC2008 | Periods | OutputBasicPrices_1 | IntermediateConsumption_2 | GrossValueAddedBasicPrices_3 |
---|---|---|---|---|---|
0 | 300025 | 1969JJ00 | 100408 | 50951 | 49457 |
1 | 300025 | 1970JJ00 | 114420 | 58772 | 55648 |
2 | 300025 | 1971JJ00 | 128110 | 65307 | 62803 |
3 | 300025 | 1972JJ00 | 139553 | 68681 | 70872 |
4 | 300025 | 1973JJ00 | 160025 | 79055 | 80970 |
List existing tables in CBS API. This takes 1-2 minutes to complete
cbs_tables <- nsoApi::cbsODataTables(url = "https://opendata.cbs.nl/ODataCatalog/Tables")
knitr::kable(cbs_tables[1:n_table, ])
Identifier | Title | Frequency | Period |
---|---|---|---|
82010NED | Zeggenschap bedrijven in Nederland; banen en lonen, bedrijfsgrootte | Perjaar | 2008 t/m 2011 |
82011NED | Zeggenschap bedrijven in Nederland; banen en lonen, bedrijfstak (SBI 2008) | Perjaar | 2008 t/m 2011 |
81179ned | Zeggenschap bedrijven in Nederland; banen en lonen (SBI’93) 2006-2008 | Stopgezet | 2006-2008 |
81251ned | Banen werknemers en afstand woon-werk; woon- en werkregio’s | Perjaar | 2006 - 2014 |
80339ned | Banen van werknemers; bedrijfsgrootte en economische activiteit, 2006-2009 | Stopgezet | 2006 - 2009 |
## write.table(cbs_tables, file = "~/Downloads/cbs-tables.tsv", sep = "\t", row.names = FALSE)
api.param.datenexport <- list(
method = "DatenExport",
kennung = apiKey$GENESIS$kennung,
passwort = apiKey$GENESIS$passwort,
namen = "81000BJ002",
bereich = "oeffentlich",
format = "csv",
werte = "false",
metadaten = "false",
zusatz = "true",
startjahr = "2013",
endjahr = "2014",
zeitscheiben = "",
regionalschluessel = "",
sachmerkmal = "",
sachschluessel = "",
stand = "01.01.2001",
sprache = "de"
)
## retrieve metadata
xml_list_datenexport_meta <- genesisAPI(api.param = api.param.datenexport,
service = "ExportService",
curl = curl)
meta_df <- genesisXMLtoDF(xml.list = xml_list_datenexport_meta,
meta = TRUE)
knitr::kable(meta_df)
|
|
api.param.datenexport <- list(
method = "DatenExport",
kennung = apiKey$GENESIS$kennung,
passwort = apiKey$GENESIS$passwort,
namen = "81000BJ002",
bereich = "oeffentlich",
format = "csv",
werte = "true",
metadaten = "false",
zusatz = "false",
startjahr = "2013",
endjahr = "2014",
zeitscheiben = "",
regionalschluessel = "",
sachmerkmal = "",
sachschluessel = "",
stand = "01.01.2001",
sprache = "de"
)
## retrieve data
xml_list_datenexport <- genesisAPI(api.param = api.param.datenexport,
service = "ExportService",
curl = curl)
## convert to data frame
data_df <- genesisXMLtoDF(xml.list = xml_list_datenexport)
knitr::kable(data_df[1:n_table,])
VGRPB5 | WZ08G2 | DINSG | ZI-WERT | UNIT | WERT |
---|---|---|---|---|---|
VGRJPM | WZ08-A | DG | 2013 | BWS001 | 24.005 |
VGRJPM | WZ08-B-18 | DG | 2013 | BWS001 | 654.188 |
VGRJPM | WZ08-C | DG | 2013 | BWS001 | 572.186 |
VGRJPM | WZ08-F | DG | 2013 | BWS001 | 113.247 |
VGRJPM | WZ08-G-01 | DG | 2013 | BWS001 | 395.395 |
genesis_tables <- nsoApi::genesisTables(
namefilter = 11, # for testing - takes a few minutes to complete
## namefilter = c(11:14, 2, 3, 41:49, 5:9),
fields = c("code", "beschriftungstext"),
kennung = apiKey$GENESIS$kennung,
passwort = apiKey$GENESIS$passwort
)
## "11*" filter: 4 entries
genesis_tables$beschriftungstext <- gsub("\n", " ", genesis_tables$beschriftungstext)
knitr::kable(genesis_tables)
code | beschriftungstext |
---|---|
11111BJ001 | Feststellung des Gebietsstands, Gebietsflche, Deutschland insgesamt, Stichtag |
11111KJ001 | Feststellung des Gebietsstands, Gebietsflche, Kreise, Stichtag |
11111LJ001 | Feststellung des Gebietsstands, Gebietsflche, Bundeslnder, Stichtag |
NA | NA |
The provider has been taken offline
api.param <- list(context = "Census",
geog = "2011WARDH",
totals = "false",
apikey = "xPuqnMzZ01")
ons_data <- onsCsvData(api.param = api.param, dataset = "QS208EW")
knitr::kable(ons_data[1:n_table, 1:7])
ons_tables <- onsAPI(method = "contexts", api.key = apiKey$ONS$apikey)
knitr::kable(ons_tables)
ons_collections <- onsAPI(method = "collections", context = "Economic", api.key = apiKey$ONS$apikey)
titles <- sapply(ons_collections$name[, 1], function(x) x[1, 2])
titles <- iconv(titles, "latin1", "ASCII", sub="")
ons_collections_df <- data.frame(ID = ons_collections$id, Title = titles)
knitr::kable(ons_collections_df[1:n_table, ])
dataset <- "OGD_vgr001_VGRJahresR_1"
statat_data <- statatAPI(dataset = dataset)
## Warning: attributes are not identical across measure variables; they will
## be dropped
knitr::kable(statat_data[1:n_table, ])
date | C_VGRW04_0 | variable | value |
---|---|---|---|
1976-01-01 | VGRW04-1 | F_ISIS_4575 | 41853.8 |
1976-01-01 | VGRW04-2 | F_ISIS_4575 | 14974.9 |
1976-01-01 | VGRW04-3 | F_ISIS_4575 | 16556.6 |
1976-01-01 | VGRW04-4 | F_ISIS_4575 | 17803.1 |
1976-01-01 | VGRW04-5 | F_ISIS_4575 | -150.7 |
df <- statatTables()
knitr::kable(df[1:n_table, ])
ID | Title |
---|---|
OGD_konjidxhan10_Konjunktur_Handel_1 | Konjunkturindizes Handel Basis 2010 |
OGD_f1531neu_Aussenhandel_1 | Außenhandelsdaten Jährlich |
OGD_ahlgrhs_Aussenhandel_3 | Außenhandelsdaten Jährlich nach Ländergruppen |
OGD_ahlgrhs_Aussenhandel_2 | Außenhandelsdaten Jährlich nach SITC |
OGD_touextsai_Tour_HKL_1 | Nächtigungsstatistik ab November 1973 - Nächtigungen nach Herkunftsländern und Bundesländern |
Obtaining a list of tables for Finland is tedious due to the hierarchical structure of the API. At each of the four levels, we must call pxweb::get_pxweb_levels
and interatively concstruct the tree of datasets. This tree is used as input when retrieving the datasets. We must know the path to a dataset a priori.
pxnet2_tables <- read.table(system.file("extdata", "content", "pxnet2_tables.tsv", package = "nsoApi"), header = TRUE)
knitr::kable(pxnet2_tables[1:10,])
ID | Title | Path |
---|---|---|
010_asas_tau_101.px | Household-dwelling units by number of persons and type of building 1985-2014 | asu/asas |
020_asas_tau_102.px | Household-dwelling units and dwelling population by number of persons, by number of rooms and type of building 2005-2014 | asu/asas |
022_asas_tau_106.px | Household-dwelling units and housing population by housing density 1989-2014 | asu/asas |
030_asas_tau_103.px | Household-dwelling units by number of persons and type of building and age and sex of the oldest person 2005-2014 | asu/asas |
040_asas_tau_104.px | Household-dwelling units by number of persons and tenure status 2005-2013 | asu/asas |
050_asas_tau_105.px | Dwellings by type of building, occupansy and year of construction 31.12.2014 | asu/asas |
010_ashi_tau_101.px | Indices of prices of existing dwellings 2010=100, quarterly | asu/ashi |
020_ashi_tau_102.px | Distributions of prices per square meter of existing dwellings since 2010 | asu/ashi |
030_ashi_tau_103.px | Indices of prices of dwellings, 1970=100, 1983=100, 2000=100 and 2005=100 | asu/ashi |
040_ashi_tau_104.px | Prices of existing dwellings by postal code and year of construction | asu/ashi |
flow <- "070_vtp_tau_071.px"
baseURL <- "http://pxnet2.stat.fi/PXWEB/api/v1/en/StatFin"
path <- as.character(pxnet2_tables$Path[pxnet2_tables$ID==flow])
req.uri <- file.path(baseURL, path, flow)
bottom_node <- pxweb::get_pxweb_metadata(req.uri)
dims <- pxweb::get_pxweb_dims(bottom_node)
## dims_list <- as.list(rep("*", length(names(dims))))
## names(dims_list) <- names(dims)
dims_list <- lapply(dims, function(x) sample(x$values, 3))
pxweb_data <- # retrieve data
pxweb::get_pxweb_data(
url = req.uri,
dims = dims_list,
clean = TRUE)
knitr::kable(pxweb_data[1:10,])
Industry | Sector | Transaction | Information | Year | values |
---|---|---|---|---|---|
0 Industries total | S129 Pension funds | P2K Intermediate consumption | Current prices | 1982 | NA |
0 Industries total | S129 Pension funds | P2K Intermediate consumption | Current prices | 2000 | 5.0 |
0 Industries total | S129 Pension funds | P2K Intermediate consumption | Current prices | 2003 | 7.0 |
0 Industries total | S129 Pension funds | P2K Intermediate consumption | At year 2010 prices | 1982 | NA |
0 Industries total | S129 Pension funds | P2K Intermediate consumption | At year 2010 prices | 2000 | 5.0 |
0 Industries total | S129 Pension funds | P2K Intermediate consumption | At year 2010 prices | 2003 | 7.0 |
0 Industries total | S129 Pension funds | P2K Intermediate consumption | Changes in volume indices, % | 1982 | NA |
0 Industries total | S129 Pension funds | P2K Intermediate consumption | Changes in volume indices, % | 2000 | NA |
0 Industries total | S129 Pension funds | P2K Intermediate consumption | Changes in volume indices, % | 2003 | 16.7 |
0 Industries total | S129 Pension funds | P51CK Consumption of fixed capital | Current prices | 1982 | NA |
Same as Statistics Finland, SCB is using the pxweb standard for their API.
scb_tables <- read.table(system.file("extdata", "content", "scb_tables.tsv", package = "nsoApi"), header = TRUE)
knitr::kable(scb_tables[1:10,])
ID | Title | Path |
---|---|---|
LonArb07Privat | Average hourly earnings of manual workers in the private sector (KLP) by industrial classification SNI2007 and overtime pay. Month 2008M01 - 2015M05 | AM/AM0101/AM0101A |
LonTjanst07Privat | Average monthly salary of non-manual workers in the private sector (KLP) by industrial classification SNI2007 and variable supplements. Month 2008M01 - 2015M05 | AM/AM0101/AM0101B |
LonTjanstPrivat | Average monthly salary of non-manual workers in the private sector (KLP) by industrial classification SNI2002 and variable supplements (no update). Month 1996M01 - 2008M12 | AM/AM0101/AM0101C |
LonArbPrivat | Average hourly earnings of manual workers in the private sector (KLP) by industrial classification SNI2002 and overtime pay (no update). Month 1996M01 - 2008M12 | AM/AM0101/AM0101C |
KLStabell14LpMan | Average monthly salary for monthly paid and number of employees in the governmental sector (KLS) by authority, sex and full-/ part-time. Month 2010M01 - 2015M05 | AM/AM0102/AM0102A |
KLStab352007LpMan | Average monthly salary for monthly paid and number of employees in the governmental sector (KLS) by industry sector (SNI2007), sex and full-/part-time. Month 2010M01 - 2015M05 | AM/AM0102/AM0102A |
KLStab352007 | Average monthly salary and number of employees in the governmental sector (KLS) by industry sector (SNI2007), sex and full-/part-time. Month 2009M01 - 2011M12 | AM/AM0102/AM0102B |
KLStab35 | Average monthly salary and number of employees in the governmental sector (KLS) by industry sector (SNI92), sex and full-/part-time. Month 2000M01 - 2008M12 | AM/AM0102/AM0102B |
KLStabell14 | Average monthly salary and number of employees in the governmental sector (KLS) by authority, sex and full-/part-time. Month 2000M01 - 2011M12 | AM/AM0102/AM0102B |
SLP9a07 | Average hourly pay, manual workers private sector (SLP) by pay element by industry SNI 2007, type of working time and sex. Year 2008 - 2014 | AM/AM0103/AM0103A |
baseURL <- "http://api.scb.se/OV0104/v1/doris/en/ssd"
flow <- "NR0103ENS2010T08A"
path <- as.character(scb_tables$Path[scb_tables$ID==flow])
req.uri <- file.path(baseURL, path, flow)
bottom_node <- pxweb::get_pxweb_metadata(req.uri)
dims <- pxweb::get_pxweb_dims(bottom_node)
## dims_list <- as.list(rep("*", length(names(dims))))
## names(dims_list) <- names(dims)
dims_list <- lapply(dims, function(x) sample(x$values, min(length(x$values), 3)))
pxweb_data <- # retrieve data
pxweb::get_pxweb_data(
url = req.uri,
dims = dims_list,
clean = TRUE)
knitr::kable(pxweb_data[1:10,])
industrial classification NACE Rev. 2 | transaction item | year | observations | values |
---|---|---|---|---|
A01 crop and animal production, hunting and related service activities | value added at basic prices | 1999 | Current prices, SEK million | 15831 |
A01 crop and animal production, hunting and related service activities | value added at basic prices | 2000 | Current prices, SEK million | 15628 |
A01 crop and animal production, hunting and related service activities | value added at basic prices | 2002 | Current prices, SEK million | 17098 |
A01 crop and animal production, hunting and related service activities | other taxes on production (-) | 1999 | Current prices, SEK million | 2271 |
A01 crop and animal production, hunting and related service activities | other taxes on production (-) | 2000 | Current prices, SEK million | 2714 |
A01 crop and animal production, hunting and related service activities | other taxes on production (-) | 2002 | Current prices, SEK million | 3109 |
A01 crop and animal production, hunting and related service activities | wages and salaries | 1999 | Current prices, SEK million | 6004 |
A01 crop and animal production, hunting and related service activities | wages and salaries | 2000 | Current prices, SEK million | 6363 |
A01 crop and animal production, hunting and related service activities | wages and salaries | 2002 | Current prices, SEK million | 6852 |
I55-I56 hotels and restaurants | value added at basic prices | 1999 | Current prices, SEK million | 29507 |
In the latest version, the SSB API is compatible with the pxweb standard.
ssb_tables <- read.table(system.file("extdata", "content", "ssb_tables.tsv", package = "nsoApi"), header = TRUE)
knitr::kable(ssb_tables[1:10,])
ID | Title | Statistic |
---|---|---|
StatResNav02 | 07534: StatRes. The Norwegian Labour and Welfare Service. Activities and services. Members of the National Insurance Scheme, by type of benefit (closed series) 2006 - 2008 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StatResNav02 |
StatResNav1 | 07527: StatRes. The Norwegian Labour and Welfare Service (closed series) 2006 - 2013 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StatResNav1 |
StatResNav27 | 08334: StatRes. The Norwegian Labour and Welfare Service. Related indicators. Population aged 15-74, by labour force status (closed series) 2006 - 2013 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StatResNav27 |
StatResNav28 | 09039: StatRes. The Norwegian Labour and Welfare Service. Registered unemployment (NAV-figures) (C) (closed series) 2006 - 2013 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StatResNav28 |
StatResNav29 | 09051: StatRes. The Norwegian Labour and Welfare Service. People on unemployment measures (C) (closed series) 2006 - 2013 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StatResNav29 |
StatResNav30 | 09052: StatRes. The Norwegian Labour and Welfare Service. Recipients of unemployment measures (C) (closed series) 2006 - 2013 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StatResNav30 |
StonadTrygd | 08367: StatRes. Transfers distributed, by various benefits and social security (NOK million) (closed series) 2006 - 2013 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StonadTrygd |
StatResNav32 | 09054: StatRes. The Norwegian Labour and Welfare Service. People with reduced working capacity on unemployment measures (C) (closed series) 2006 - 2013 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StatResNav32 |
StatResNav33 | 09055: StatRes. The Norwegian Labour and Welfare Service. Recipients of work assessment allowance (C) (closed series) 2010 - 2013 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StatResNav33 |
StatResNav34 | 09056: StatRes. The Norwegian Labour and Welfare Service. Recipients of disability pension (C) (closed series) 2006 - 2013 | http://data.ssb.no/api/v0/en/table/al/al03/nav_statres/StatResNav34 |
baseURL <- "http://data.ssb.no/api/v0"
## baseURL <- "http://data.ssb.no/api/v0/en/table"
flow <- "OrgsatEksUlonn"
req.uri <- as.character(ssb_tables$Statistic[ssb_tables$ID==flow])[1]
## req.uri <- file.path(baseURL, path, flow)
bottom_node <- pxweb::get_pxweb_metadata(req.uri)
dims <- pxweb::get_pxweb_dims(bottom_node)
## dims_list <- as.list(rep("*", length(names(dims))))
## names(dims_list) <- names(dims)
dims_list <- lapply(dims, function(x) sample(x$values, min(3, length(x$values))))
pxweb_data <- # retrieve data
pxweb::get_pxweb_data(
url = req.uri,
dims = dims_list,
clean = TRUE)
knitr::kable(pxweb_data[1:10,])
main indicator | activity (ICNPO) | year | contents | values |
---|---|---|---|---|
Output | 00 Total | 2007 | Current prices (NOK million) | 67774.0 |
Output | 00 Total | 2009 | Current prices (NOK million) | 78814.0 |
Output | 00 Total | 2014 | Current prices (NOK million) | 103001.0 |
Output | 00 Total | 2007 | Annual change (per cent) | 7.9 |
Output | 00 Total | 2009 | Annual change (per cent) | 7.0 |
Output | 00 Total | 2014 | Annual change (per cent) | 6.1 |
Output | 04000c Social services. Substance abuse-related care | 2007 | Current prices (NOK million) | 1120.0 |
Output | 04000c Social services. Substance abuse-related care | 2009 | Current prices (NOK million) | 1280.0 |
Output | 04000c Social services. Substance abuse-related care | 2014 | Current prices (NOK million) | 1518.0 |
Output | 04000c Social services. Substance abuse-related care | 2007 | Annual change (per cent) | 16.7 |