Skill Tracer
Skills › Scientific Agent Skills › market-research-reports › official_data_sources.md

official_data_sources.md

207 lines · skills/market-research-reports/references/official_data_sources.md at 5f2bc03

Open on GitHub
1# Official Data and Filing Sources
2
3Reviewed against first-party guidance on 2026-10-01. API rules can change:
4recheck the linked terms and limits before automated or high-volume use. The
5bundled scripts do not call these services and do not require API keys.
6This is a routing and metadata guide, not an executed API client: documentation
7review does not establish authenticated retrieval or current service availability.
8

Official Data and Filing Sources

Reviewed against first-party guidance on 2026-10-01. API rules can change: recheck the linked terms and limits before automated or high-volume use. The bundled scripts do not call these services and do not require API keys. This is a routing and metadata guide, not an executed API client: documentation review does not establish authenticated retrieval or current service availability.

Raw lines 1 to 8
9## Routing by claim
11Start with the original authority most fit for the claim:
13- company financials and risk disclosures: the jurisdiction's official filing
14 system, then the filed document;
15- establishment, employment, prices, production, trade, population, and GDP:
16 the responsible national statistical office or central bank;
17- rules, approvals, enforcement, licenses, and consultations: the responsible
18 regulator or official legal gazette;
19- cross-country indicators: the original national source when comparability is
20 not required; otherwise an international harmonized dataset with metadata;
21- classifications: the current official NAICS, NACE, ISIC, product, trade, or
22 sector taxonomy and its correspondence tables.
24Do not treat an aggregator as an independent corroborating source when it
25reproduces the same underlying series.

Routing by claim

Start with the original authority most fit for the claim:

  • company financials and risk disclosures: the jurisdiction's official filing system, then the filed document;
  • establishment, employment, prices, production, trade, population, and GDP: the responsible national statistical office or central bank;
  • rules, approvals, enforcement, licenses, and consultations: the responsible regulator or official legal gazette;
  • cross-country indicators: the original national source when comparability is not required; otherwise an international harmonized dataset with metadata;
  • classifications: the current official NAICS, NACE, ISIC, product, trade, or sector taxonomy and its correspondence tables.

Do not treat an aggregator as an independent corroborating source when it reproduces the same underlying series.

Raw lines 9 to 26
27## United States

United States

Raw lines 27 to 28
29### SEC EDGAR and company filings
31- [SEC Developer Resources](https://www.sec.gov/about/developer-resources)
32 documents company submissions and extracted XBRL data APIs.
33- [Accessing EDGAR Data](https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data)
34 requires a declared user agent and states a current maximum of 10 requests per
35 second across the user's machines. Download only what is needed.
36- EDGAR filings can be corrected or removed after acceptance. Record accession
37 number, form, filing date, reporting period, amendment status, exact table or
38 XBRL fact, units, and retrieval date.
39- Consolidated company revenue is not automatically market revenue. Remove
40 out-of-scope products/geographies and avoid summing parent/subsidiary or
41 channel/end-customer values twice.

SEC EDGAR and company filings

  • SEC Developer Resources documents company submissions and extracted XBRL data APIs.
  • Accessing EDGAR Data requires a declared user agent and states a current maximum of 10 requests per second across the user's machines. Download only what is needed.
  • EDGAR filings can be corrected or removed after acceptance. Record accession number, form, filing date, reporting period, amendment status, exact table or XBRL fact, units, and retrieval date.
  • Consolidated company revenue is not automatically market revenue. Remove out-of-scope products/geographies and avoid summing parent/subsidiary or channel/end-customer values twice.
Raw lines 29 to 42
43### U.S. Census Bureau
45- The [Census Data API User Guide](https://www.census.gov/data/developers/guidance/api-user-guide.html)
46 links data to a geographic boundary and a dataset vintage.
47- As revised 2026-05-14, the
48 [query-limits page](https://www.census.gov/data/developers/guidance/api-user-guide.Query_Limits.html)
49 permits up to 50 variables per query and requires a key for all data queries.
50 The [key page](https://www.census.gov/data/developers/guidance/api-user-guide.API_Key.html)
51 describes free registration. The key travels in the query URL; redact `key`
52 before storing source URLs, logs, or request provenance. Never place a key in a
53 report, source ledger, or bundled script.
54- Use program-specific methodology, margins of error, universe, geography, and
55 vintage. ACS estimates, Population Estimates, and decennial counts are not
56 interchangeable.
57- The [2022 Economic Census methodology](https://www.census.gov/programs-surveys/economic-census/year/2022/technical-documentation/methodology.html)
58 defines the target population, sampling frame, exclusions, administrative
59 data, imputation, disclosure avoidance, and product collection.
60- The [NAICS site](https://www.census.gov/naics) identifies 2022 NAICS as the
61 current published structure while a 2027 revision process is underway. Store
62 the version used.

U.S. Census Bureau

  • The Census Data API User Guide links data to a geographic boundary and a dataset vintage.
  • As revised 2026-05-14, the query-limits page permits up to 50 variables per query and requires a key for all data queries. The key page describes free registration. The key travels in the query URL; redact key before storing source URLs, logs, or request provenance. Never place a key in a report, source ledger, or bundled script.
  • Use program-specific methodology, margins of error, universe, geography, and vintage. ACS estimates, Population Estimates, and decennial counts are not interchangeable.
  • The 2022 Economic Census methodology defines the target population, sampling frame, exclusions, administrative data, imputation, disclosure avoidance, and product collection.
  • The NAICS site identifies 2022 NAICS as the current published structure while a 2027 revision process is underway. Store the version used.
Raw lines 43 to 63
64### Bureau of Labor Statistics
66The [BLS API FAQ](https://www.bls.gov/developers/api_faqs.htm), last modified
672023-08-30, documents:
69- registered v2: 500 queries/day, 50 series/query, 20 years/query;
70- unregistered v1: 25 queries/day, 25 series/query, 10 years/query;
71- both: 50 requests per 10 seconds;
72- v2 registration renewal at least annually;
73- v1 returns observations and footnotes without descriptive metadata.
75Record series ID, survey/program, seasonal adjustment, units, frequency,
76footnotes, publication date, and revision status. Consult the program's
77methodology and release calendar rather than relying on the API response alone.
78Inspect `message` and each series' observations even when the response says
79`REQUEST_SUCCEEDED`: the FAQ shows invalid-series messages with that status.

Bureau of Labor Statistics

The BLS API FAQ, last modified 2023-08-30, documents:

  • registered v2: 500 queries/day, 50 series/query, 20 years/query;
  • unregistered v1: 25 queries/day, 25 series/query, 10 years/query;
  • both: 50 requests per 10 seconds;
  • v2 registration renewal at least annually;
  • v1 returns observations and footnotes without descriptive metadata.

Record series ID, survey/program, seasonal adjustment, units, frequency, footnotes, publication date, and revision status. Consult the program's methodology and release calendar rather than relying on the API response alone. Inspect message and each series' observations even when the response says REQUEST_SUCCEEDED: the FAQ shows invalid-series messages with that status.

Raw lines 64 to 80
81### Bureau of Economic Analysis
83The [BEA API User Guide](https://apps.bea.gov/api/_pdf/bea_web_service_api_user_guide.pdf),
84dated 2026-04-20, requires a registered UserID and documents three rolling
85per-minute limits:
87- 100 requests;
88- 100 MB retrieved;
89- 30 errors.
91BEA returns HTTP 429 and a `Retry-After` header when throttled. The guide warns
92that limits may change. Query metadata methods before data, restrict years and
93dimensions, and avoid broad `ALL` requests. Record current versus chained
94dollars, reference year, table/line code, frequency, seasonality, and release
95vintage. Chained-dollar components may not be additive; use published
96contributions or current-dollar shares where appropriate.
97Check `BEAAPI.Results.Error` as well as HTTP status; request parameters may be
98echoed in the response, so redact `UserID` before archiving it.

Bureau of Economic Analysis

The BEA API User Guide, dated 2026-04-20, requires a registered UserID and documents three rolling per-minute limits:

  • 100 requests;
  • 100 MB retrieved;
  • 30 errors.

BEA returns HTTP 429 and a Retry-After header when throttled. The guide warns that limits may change. Query metadata methods before data, restrict years and dimensions, and avoid broad ALL requests. Record current versus chained dollars, reference year, table/line code, frequency, seasonality, and release vintage. Chained-dollar components may not be additive; use published contributions or current-dollar shares where appropriate. Check BEAAPI.Results.Error as well as HTTP status; request parameters may be echoed in the response, so redact UserID before archiving it.

Raw lines 81 to 99
100### Federal Reserve and FRED/ALFRED
102- [FRED API documentation](https://fred.stlouisfed.org/docs/api/fred/) describes
103 v2 bulk release history and v1 series-level FRED/ALFRED access.
104- A registered key is required under the
105 [FRED API Terms](https://fred.stlouisfed.org/docs/api/terms_of_use.html).
106 The reviewed terms do not state one fixed numerical request ceiling; they
107 reserve the right to impose or change limits.
108- The terms warn that third parties may own series and impose additional
109 restrictions. Follow the original producer's rights and attribution.
110- FRED normally presents latest values; ALFRED preserves real-time vintages.
111 Record source, release, series ID, frequency, units, seasonal adjustment,
112 notes, and vintage dates.

Federal Reserve and FRED/ALFRED

  • FRED API documentation describes v2 bulk release history and v1 series-level FRED/ALFRED access.
  • A registered key is required under the FRED API Terms. The reviewed terms do not state one fixed numerical request ceiling; they reserve the right to impose or change limits.
  • The terms warn that third parties may own series and impose additional restrictions. Follow the original producer's rights and attribution.
  • FRED normally presents latest values; ALFRED preserves real-time vintages. Record source, release, series ID, frequency, units, seasonal adjustment, notes, and vintage dates.
Raw lines 100 to 113
114## International and harmonized sources

International and harmonized sources

Raw lines 114 to 115
116### World Bank
118The [Indicators API documentation](https://datahelpdesk.worldbank.org/knowledgebase/articles/889392-about-the-indicators-api-documentation)
119states that v2 is current, v1 is discontinued, and API authentication is not
120required. Preserve indicator code, database/source, source note, source
121organization, unit, income/region classification vintage, and retrieval date.
122The [WDI catalog](https://datacatalog.worldbank.org/search/dataset/0037712/world-development-indicators)
123publishes metadata and revision-history resources. A World Bank indicator may
124originate with a national agency or another international organization; retain
125that lineage.

World Bank

The Indicators API documentation states that v2 is current, v1 is discontinued, and API authentication is not required. Preserve indicator code, database/source, source note, source organization, unit, income/region classification vintage, and retrieval date. The WDI catalog publishes metadata and revision-history resources. A World Bank indicator may originate with a national agency or another international organization; retain that lineage.

Raw lines 116 to 126
127### International Monetary Fund
129The [IMF Data API page](https://data.imf.org/en/Resource-Pages/IMF-API) states
130that IMF data are available through SDMX 2.1 and SDMX 3.0 APIs. Use the
131dataset's data structure, codelists, unit, scale, frequency, observation status,
132and methodological metadata. Do not assume similarly named indicators across
133IMF datasets have identical definitions.
134Use the current API portal linked on that page (its Swagger explorer requires
135portal sign-in) to select a data service and verify its access requirements.
136The separate SDMX Central guide concerns structure, validation, conversion, and
137registration services; it does not establish a data-download endpoint contract.

International Monetary Fund

The IMF Data API page states that IMF data are available through SDMX 2.1 and SDMX 3.0 APIs. Use the dataset's data structure, codelists, unit, scale, frequency, observation status, and methodological metadata. Do not assume similarly named indicators across IMF datasets have identical definitions. Use the current API portal linked on that page (its Swagger explorer requires portal sign-in) to select a data service and verify its access requirements. The separate SDMX Central guide concerns structure, validation, conversion, and registration services; it does not establish a data-download endpoint contract.

Raw lines 127 to 138
139### OECD
141The [OECD Data Explorer API guide](https://www.oecd.org/en/data/insights/data-explainers/2024/09/api.html),
142published 2025-04-30, describes the SDMX API, free access subject to OECD terms,
143and rate limiting without publishing one universal numerical ceiling on that
144page. It warns that omitting a dataflow version selects the latest version and
145that later structures may not be backward compatible. Store agency, dataset,
146dataflow version, dimensions, codes, attributes, and query.

OECD

The OECD Data Explorer API guide, published 2025-04-30, describes the SDMX API, free access subject to OECD terms, and rate limiting without publishing one universal numerical ceiling on that page. It warns that omitting a dataflow version selects the latest version and that later structures may not be backward compatible. Store agency, dataset, dataflow version, dimensions, codes, attributes, and query.

Raw lines 139 to 147
148### Eurostat
150The [Eurostat API introduction](https://ec.europa.eu/eurostat/web/user-guides/data-browser/api-data-access/api-introduction)
151documents Statistics, SDMX 2.1, SDMX 3.0, catalogue, and asynchronous services.
152It states that datasets are updated twice daily when changes are available and
153that the database contains only the latest version, without past-version
154documentation. Preserve the retrieved file and update timestamp when a
155reproducible vintage matters.
157The [ESS quality and metadata handbook](https://ec.europa.eu/eurostat/web/products-manuals-and-guidelines/-/ks-gq-21-021)
158is a standard for reporting source, process, quality, and metadata. Record flags,
159breaks, seasonal adjustment, units, NUTS/geography version, and dataset code.

Eurostat

The Eurostat API introduction documents Statistics, SDMX 2.1, SDMX 3.0, catalogue, and asynchronous services. It states that datasets are updated twice daily when changes are available and that the database contains only the latest version, without past-version documentation. Preserve the retrieved file and update timestamp when a reproducible vintage matters.

The ESS quality and metadata handbook is a standard for reporting source, process, quality, and metadata. Record flags, breaks, seasonal adjustment, units, NUTS/geography version, and dataset code.

Raw lines 148 to 160
161### Other national statistical agencies
163Use the relevant country's official agency before secondary compilations.
164Examples of current first-party interfaces:
166- [Statistics Canada Web Data Service](https://www.statcan.gc.ca/en/developers/wds/user-guide)
167 provides data and metadata. Guide revision 1.6 (2025-01-24) documents a
168 50-request/second server limit and 25-request/second individual IP limit, and
169 possible HTTP 409 responses during update windows.
170- The [UK ONS Developer Hub](https://developer.ons.gov.uk/) describes an open,
171 no-key beta API. It warns that breaking changes can occur; store dataset,
172 edition, and version because new versions reflect corrections, revisions, or
173 new data.
175For any country, confirm:
1771. the official statistics producer and legal mandate;
1782. release calendar, methodology, quality statement, and revision policy;
1793. classification and geography versions;
1804. API/download terms and current operational limits;
1815. whether the dataset is official, experimental, modeled, or an administrative
182 extract.

Other national statistical agencies

Use the relevant country's official agency before secondary compilations. Examples of current first-party interfaces:

  • Statistics Canada Web Data Service provides data and metadata. Guide revision 1.6 (2025-01-24) documents a 50-request/second server limit and 25-request/second individual IP limit, and possible HTTP 409 responses during update windows.
  • The UK ONS Developer Hub describes an open, no-key beta API. It warns that breaking changes can occur; store dataset, edition, and version because new versions reflect corrections, revisions, or new data.

For any country, confirm:

  1. the official statistics producer and legal mandate;
  2. release calendar, methodology, quality statement, and revision policy;
  3. classification and geography versions;
  4. API/download terms and current operational limits;
  5. whether the dataset is official, experimental, modeled, or an administrative extract.
Raw lines 161 to 183
184## Industry and product classifications
186- [NAICS](https://www.census.gov/naics) classifies establishments by primary
187 economic activity; it is not itself a product-market definition.
188- [NACE Rev. 2.1](https://ec.europa.eu/eurostat/web/products-eurostat-news/w/wdn-20250624-1)
189 began feeding European statistics from 2025. Use the 2025 manual and
190 correspondence tables when comparing Rev. 2 and Rev. 2.1.
191- Product classifications (NAPCS, CPA, PRODCOM, CPC, HS/CN) may fit market
192 outputs better than an establishment-based industry code.
193- A concordance can be one-to-many or many-to-many. Never apply it as a
194 lossless conversion without weights and uncertainty.

Industry and product classifications

  • NAICS classifies establishments by primary economic activity; it is not itself a product-market definition.
  • NACE Rev. 2.1 began feeding European statistics from 2025. Use the 2025 manual and correspondence tables when comparing Rev. 2 and Rev. 2.1.
  • Product classifications (NAPCS, CPA, PRODCOM, CPC, HS/CN) may fit market outputs better than an establishment-based industry code.
  • A concordance can be one-to-many or many-to-many. Never apply it as a lossless conversion without weights and uncertainty.
Raw lines 184 to 195
196## API handling rules
198- Use APIs only during research, with user-approved network access.
199- Keep credentials outside reports and scripts; never commit keys.
200- Respect official terms, user-agent requirements, rate limits, retries, and
201 bulk-download guidance.
202- Cache lawful downloads, record the exact query and retrieval time, and avoid
203 repeatedly requesting unchanged data.
204- Validate response status, metadata, units, flags, suppression, and missing
205 values before analysis.
206- Treat current limits in this file as a dated snapshot, not a permanent
207 entitlement.

API handling rules

  • Use APIs only during research, with user-approved network access.
  • Keep credentials outside reports and scripts; never commit keys.
  • Respect official terms, user-agent requirements, rate limits, retries, and bulk-download guidance.
  • Cache lawful downloads, record the exact query and retrieval time, and avoid repeatedly requesting unchanged data.
  • Validate response status, metadata, units, flags, suppression, and missing values before analysis.
  • Treat current limits in this file as a dated snapshot, not a permanent entitlement.
Raw lines 196 to 207