Skill Tracer
Skills › Marketing Skills › reverse-engineer-marketing › data-sources.md

data-sources.md

142 lines · skills/reverse-engineer-marketing/references/data-sources.md at 88c79f6

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Data Sources for a Teardown
2
3Where each stage's evidence comes from. The skill is tool-agnostic for search and traffic data: use whatever the user has, but keep one vendor per curve. Social posts come from ScrapeCreators or Apify. Free sources go first. Quote a paid call's cost before making it, and keep a running total against the cap.
4

Data Sources for a Teardown

Where each stage's evidence comes from. The skill is tool-agnostic for search and traffic data: use whatever the user has, but keep one vendor per curve. Social posts come from ScrapeCreators or Apify. Free sources go first. Quote a paid call's cost before making it, and keep a running total against the cap.

Raw lines 1 to 4
5## Contents
6- The bundled script
7- Free sources
8- Search, traffic, and backlinks
9- AI mentions and citations
10- Social posts and ad libraries
11- Transcripts
12- Browser and inbox for going inside
13- The user's own data
14- When a source fails
16---

Contents

  • The bundled script
  • Free sources
  • Search, traffic, and backlinks
  • AI mentions and citations
  • Social posts and ad libraries
  • Transcripts
  • Browser and inbox for going inside
  • The user's own data
  • When a source fails
Raw lines 5 to 17
18## The Bundled Script
20`scripts/teardown.mjs` (Node 18+, no dependencies, no API keys) covers the free evidence:
22| Command | What it returns |
23|---|---|
24| `site <domain>` | Every sitemap URL, grouped by page family, with locale prefixes folded in |
25| `stack <domain> --months 24` | Ad pixels, analytics, testing, referral, and lifecycle scripts, with the first month each appears in Wayback's monthly captures |
26| `history <url> --months 24` | Months a page's listed prices changed, with the archive link for each |
27| `hero <url> --months 24` | Months the homepage headline, title, or meta description changed, with archive links |
28| `sitemap-growth <domain> --months 24` | Sitemap size by month, and when each page family first appeared |
29| `dns <domain>` | Mail tools (SPF, MX), domain verifications, and the platforms behind common marketing subdomains |
30| `subdomains <domain>` | Subdomains from certificate logs with first-seen dates, preview and staging names filtered out |
31| `ads <name> --domain <domain>` | Links to the Meta, Google, LinkedIn, and TikTok ad libraries for the company |
32| `sc-ads <name> --domain <domain>` | Active Meta, Google, and TikTok ads through ScrapeCreators (needs `SCRAPECREATORS_API_KEY`; `--dry-run` shows the requests) |
33| `detect <domain>` | Likely industry packs with the reasons (scripts, page text, schema types) |
34| `shopify <domain>` | Public Shopify catalog: products with created dates, prices and sale flags, bestseller order, installed apps |
35| `experiments <domain>` | Client-side testing tools, plus readable Optimizely and GrowthBook config names |
36| `phones <url>` | Phone numbers in the page, call-tracking tools, and UTM test URLs for a browser check |
37| `filings <company>` | SEC filings for public companies and Form D fundraise filings (needs a contact in `--ua` or `SEC_USER_AGENT`) |
38| `cfpb <company>` | CFPB consumer complaints by month, with top products and issues |
39| `fdic <bank>` | FDIC quarterly assets, deposits, and employee counts |
40| `moves` | Lift per cluster of moves and leading channels (stage 7) |
41| `diff` | Moves in a new log that weren't in the old one (Watch mode) |
43Limits to state in the method notes:
44- `stack` only sees scripts in the page source. Tags loaded through a tag manager or after a consent banner won't show, so a missing pixel isn't proof the channel wasn't used.
45- `stack`'s first month is the first monthly capture with the script. If it equals the window's start month (`present_at_start`), the tool was already there.
46- When a site blocks scripted fetches, `detect`, `experiments`, and `phones` read its latest Wayback capture instead and say so in `source`. The script doesn't disguise itself to get past a block.
47- `history` reads prices from the HTML. Pages that render prices with JavaScript show up under `no_prices`. Open those captures by hand.

The Bundled Script

scripts/teardown.mjs (Node 18+, no dependencies, no API keys) covers the free evidence:

CommandWhat it returns
site <domain>Every sitemap URL, grouped by page family, with locale prefixes folded in
stack <domain> --months 24Ad pixels, analytics, testing, referral, and lifecycle scripts, with the first month each appears in Wayback's monthly captures
history <url> --months 24Months a page's listed prices changed, with the archive link for each
hero <url> --months 24Months the homepage headline, title, or meta description changed, with archive links
sitemap-growth <domain> --months 24Sitemap size by month, and when each page family first appeared
dns <domain>Mail tools (SPF, MX), domain verifications, and the platforms behind common marketing subdomains
subdomains <domain>Subdomains from certificate logs with first-seen dates, preview and staging names filtered out
ads <name> --domain <domain>Links to the Meta, Google, LinkedIn, and TikTok ad libraries for the company
sc-ads <name> --domain <domain>Active Meta, Google, and TikTok ads through ScrapeCreators (needs SCRAPECREATORS_API_KEY; --dry-run shows the requests)
detect <domain>Likely industry packs with the reasons (scripts, page text, schema types)
shopify <domain>Public Shopify catalog: products with created dates, prices and sale flags, bestseller order, installed apps
experiments <domain>Client-side testing tools, plus readable Optimizely and GrowthBook config names
phones <url>Phone numbers in the page, call-tracking tools, and UTM test URLs for a browser check
filings <company>SEC filings for public companies and Form D fundraise filings (needs a contact in --ua or SEC_USER_AGENT)
cfpb <company>CFPB consumer complaints by month, with top products and issues
fdic <bank>FDIC quarterly assets, deposits, and employee counts
movesLift per cluster of moves and leading channels (stage 7)
diffMoves in a new log that weren't in the old one (Watch mode)

Limits to state in the method notes:

  • stack only sees scripts in the page source. Tags loaded through a tag manager or after a consent banner won't show, so a missing pixel isn't proof the channel wasn't used.
  • stack's first month is the first monthly capture with the script. If it equals the window's start month (present_at_start), the tool was already there.
  • When a site blocks scripted fetches, detect, experiments, and phones read its latest Wayback capture instead and say so in source. The script doesn't disguise itself to get past a block.
  • history reads prices from the HTML. Pages that render prices with JavaScript show up under no_prices. Open those captures by hand.
Raw lines 18 to 48
Runs code scripts/teardown.mjs :20
49## Free Sources
51| Source | Use it for |
52|---|---|
53| The target's site | Pricing, sign-up flow, docs, changelog dates, partner and affiliate pages, badges |
54| Wayback Machine | When pages and page families first appeared: `web.archive.org/cdx/search/cdx?url=target.com/templates/*&output=json&collapse=urlkey` lists archived URLs with their dates |
55| Ad libraries | Active ads, run dates, and landing pages (links from `ads`) |
56| Google Trends | Relative brand interest by month, when no brand search volume is available |
57| AI assistants | The stage 2 AI answer checks, run by hand |
58| Search operators | `"made with target"`, `site:reddit.com "target"`, `site:target.com/compare` |
59| Careers page | Open roles for the hiring check |

Free Sources

SourceUse it for
The target's sitePricing, sign-up flow, docs, changelog dates, partner and affiliate pages, badges
Wayback MachineWhen pages and page families first appeared: web.archive.org/cdx/search/cdx?url=target.com/templates/*&output=json&collapse=urlkey lists archived URLs with their dates
Ad librariesActive ads, run dates, and landing pages (links from ads)
Google TrendsRelative brand interest by month, when no brand search volume is available
AI assistantsThe stage 2 AI answer checks, run by hand
Search operators"made with target", site:reddit.com "target", site:target.com/compare
Careers pageOpen roles for the hiring check
Raw lines 49 to 60
Names a skill ads :55
61## Search, Traffic, and Backlinks
63Use whichever of these the user already has. Pick one for the curve and keep it for the whole window, and for the user's own domain in stage 8.
65| Source | Monthly history | Keywords by URL | Backlinks and anchors | Access |
66|---|---|---|---|---|
67| [DataForSEO](../../../tools/integrations/dataforseo.md) | Historical rank overview | Ranked keywords | Backlinks summary, anchors | Pay per call; CLI in the repo |
68| [Semrush](../../../tools/integrations/semrush.md) | Organic traffic history | Organic positions | Backlinks, anchors | Subscription; MCP |
69| [Ahrefs](../../../tools/integrations/ahrefs.md) | Organic traffic history | Organic keywords | Referring domains, anchors | Subscription; MCP |
70| [Similarweb](../../../tools/integrations/similarweb.md) | Visits and traffic sources by month | Top keywords | - | Subscription |
71| [RankParse](../../../tools/integrations/rankparse.md) | - | - | Backlinks, tech stack | Pay per call |
73For tech stack beyond the page source, [TheirStack](../../../tools/integrations/theirstack.md) infers tools from job posts and also covers the hiring check. For news, funding, and mentions, [Exa](../../../tools/integrations/exa.md) searches the web by meaning. Multi-provider data gateways (one key, many vendors, billed per call) are listed in the [tools registry](../../../tools/REGISTRY.md).

Search, Traffic, and Backlinks

Use whichever of these the user already has. Pick one for the curve and keep it for the whole window, and for the user's own domain in stage 8.

SourceMonthly historyKeywords by URLBacklinks and anchorsAccess
DataForSEOHistorical rank overviewRanked keywordsBacklinks summary, anchorsPay per call; CLI in the repo
SemrushOrganic traffic historyOrganic positionsBacklinks, anchorsSubscription; MCP
AhrefsOrganic traffic historyOrganic keywordsReferring domains, anchorsSubscription; MCP
SimilarwebVisits and traffic sources by monthTop keywords-Subscription
RankParse--Backlinks, tech stackPay per call

For tech stack beyond the page source, TheirStack infers tools from job posts and also covers the hiring check. For news, funding, and mentions, Exa searches the web by meaning. Multi-provider data gateways (one key, many vendors, billed per call) are listed in the tools registry.

Raw lines 61 to 74
Uses an outside tool MCP :68
75## AI Mentions and Citations
77Use the prompt tracker the user has connected. It gives share of voice and citations over time, which can go on the timeline next to traffic. Run the same prompt set for the target and the user.
79| Option | Notes |
80|---|---|
81| Prompt trackers | Peec AI, Otterly, ZipTie, LLMrefs, Profound, Omnia, Ahrefs Brand Radar, Semrush. See the monitoring tools table in **ai-seo** |
82| [Magister](https://magistermarketing.com?ref=marketingskills) | Runs AI visibility checks and citation reports across assistants from its connected workspace |
83| Manual checks | The free fallback: ask three to five buyer questions in ChatGPT, Perplexity, Gemini, and Google AI Overviews; record the order brands are named and the pages cited |
85*Disclosure: Magister is built by Corey Haines, who maintains this repository.*

AI Mentions and Citations

Use the prompt tracker the user has connected. It gives share of voice and citations over time, which can go on the timeline next to traffic. Run the same prompt set for the target and the user.

OptionNotes
Prompt trackersPeec AI, Otterly, ZipTie, LLMrefs, Profound, Omnia, Ahrefs Brand Radar, Semrush. See the monitoring tools table in ai-seo
MagisterRuns AI visibility checks and citation reports across assistants from its connected workspace
Manual checksThe free fallback: ask three to five buyer questions in ChatGPT, Perplexity, Gemini, and Google AI Overviews; record the order brands are named and the pages cited

Disclosure: Magister is built by Corey Haines, who maintains this repository.

Raw lines 75 to 86
87## Social Posts and Ad Libraries
89| Need | [ScrapeCreators](../../../tools/integrations/scrapecreators.md) | [Apify](../../../tools/integrations/apify.md) |
90|---|---|---|
91| YouTube videos and Shorts | `/v1/youtube/channel-videos`, `/v1/youtube/channel/shorts` | YouTube scraper actors |
92| TikTok videos | `/v3/tiktok/profile/videos` | TikTok scraper actors |
93| Instagram posts | `/v2/instagram/user/posts` | Instagram scraper actors |
94| X posts | `/v1/twitter/user-tweets` | X scraper actors |
95| Reddit mentions | `/v1/reddit/search` | Reddit scraper actors |
96| Meta ads | `/v1/facebook/adLibrary/company/ads` | Meta Ad Library actors |
97| Google ads | `/v1/google/company/ads` | Google Ads Transparency actors |
98| TikTok ads | `/v1/tiktok/ad-library/search` | TikTok ad actors |
99| LinkedIn posts and ads | Not used | Not used |
101- **ScrapeCreators** is one key with one response style across platforms, which makes per-channel pulls quick. Billed in credits.
102- **Apify** suits bulk pulls, scheduled runs, and platforms ScrapeCreators doesn't cover. Each actor has its own input and pricing; read its page first.
103- **LinkedIn:** read the company page, employee posts, and the LinkedIn Ad Library by hand. This repo doesn't scrape LinkedIn.
104- Media URLs expire. Download thumbnails and covers in the same run as the pull.
105- The YouTube Data API (free with a key) and Reddit's public JSON are fine fallbacks when no paid source is set up.

Social Posts and Ad Libraries

NeedScrapeCreatorsApify
YouTube videos and Shorts/v1/youtube/channel-videos, /v1/youtube/channel/shortsYouTube scraper actors
TikTok videos/v3/tiktok/profile/videosTikTok scraper actors
Instagram posts/v2/instagram/user/postsInstagram scraper actors
X posts/v1/twitter/user-tweetsX scraper actors
Reddit mentions/v1/reddit/searchReddit scraper actors
Meta ads/v1/facebook/adLibrary/company/adsMeta Ad Library actors
Google ads/v1/google/company/adsGoogle Ads Transparency actors
TikTok ads/v1/tiktok/ad-library/searchTikTok ad actors
LinkedIn posts and adsNot usedNot used
  • ScrapeCreators is one key with one response style across platforms, which makes per-channel pulls quick. Billed in credits.
  • Apify suits bulk pulls, scheduled runs, and platforms ScrapeCreators doesn't cover. Each actor has its own input and pricing; read its page first.
  • LinkedIn: read the company page, employee posts, and the LinkedIn Ad Library by hand. This repo doesn't scrape LinkedIn.
  • Media URLs expire. Download thumbnails and covers in the same run as the pull.
  • The YouTube Data API (free with a key) and Reddit's public JSON are fine fallbacks when no paid source is set up.
Raw lines 87 to 106
107## Transcripts
109For stage 4 (team voices) and the podcast-ad search in stage 5:
111| Source | Route |
112|---|---|
113| YouTube, TikTok, X video | ScrapeCreators transcript endpoints, or YouTube's own captions |
114| Podcasts | The feed's `podcast:transcript` tag, the show's YouTube upload, or a podcast search API (Listen Notes, Podscan) if the user has a key; otherwise download the audio and transcribe it locally |

Transcripts

For stage 4 (team voices) and the podcast-ad search in stage 5:

SourceRoute
YouTube, TikTok, X videoScrapeCreators transcript endpoints, or YouTube's own captions
PodcastsThe feed's podcast:transcript tag, the show's YouTube upload, or a podcast search API (Listen Notes, Podscan) if the user has a key; otherwise download the audio and transcribe it locally
Raw lines 107 to 115
116## Browser and Inbox for Going Inside
118For stage 3, with the user's approval for each action:
120| Need | Options |
121|---|---|
122| A browser the agent drives and screenshots | Claude in Chrome, agent-browser, [Browserbase](../../../tools/integrations/browserbase.md) |
123| An inbox for the email sequence | A dedicated address or alias on the user's own domain |

Browser and Inbox for Going Inside

For stage 3, with the user's approval for each action:

NeedOptions
A browser the agent drives and screenshotsClaude in Chrome, agent-browser, Browserbase
An inbox for the email sequenceA dedicated address or alias on the user's own domain
Raw lines 116 to 124
125## The User's Own Data
127For stage 8:
129| Source | What it adds |
130|---|---|
131| [Google Search Console](../../../tools/integrations/google-search-console.md) | Impressions and clicks by page family; whether the user has the target's winning families |
132| [GA4](../../../tools/integrations/ga4.md), [PostHog](../../../tools/integrations/posthog.md), [Plausible](../../../tools/integrations/plausible.md) | Channel mix and signup conversion |
133| [Google Ads](../../../tools/integrations/google-ads.md), [Meta Ads](../../../tools/integrations/meta-ads.md) | Which networks the user runs and what works there |
134| The user's social accounts | The same content pull as stage 6, for a side-by-side |
136Compare the user's real numbers only with each other. Against the target, use the same vendor's estimate for both domains.

The User's Own Data

For stage 8:

SourceWhat it adds
Google Search ConsoleImpressions and clicks by page family; whether the user has the target's winning families
GA4, PostHog, PlausibleChannel mix and signup conversion
Google Ads, Meta AdsWhich networks the user runs and what works there
The user's social accountsThe same content pull as stage 6, for a side-by-side

Compare the user's real numbers only with each other. Against the target, use the same vendor's estimate for both domains.

Raw lines 125 to 137
138## When a Source Fails
140- **Blocked page:** try one other route (another fetcher, the Wayback copy). Still blocked → list it in the method notes and move on.
141- **Endpoint error:** try one other source for the same job. Still failing → mark the item "unavailable". Never estimate it.
142- **No history:** if a tool only returns current data, say the curve is unavailable before a date. Don't stitch in another vendor.

When a Source Fails

  • Blocked page: try one other route (another fetcher, the Wayback copy). Still blocked → list it in the method notes and move on.
  • Endpoint error: try one other source for the same job. Still failing → mark the item "unavailable". Never estimate it.
  • No history: if a tool only returns current data, say the curve is unavailable before a date. Don't stitch in another vendor.
Raw lines 138 to 142