Skills › Marketing Skills › conversion-tracking › evals.json
evals.json
264 lines · skills/conversion-tracking/evals/evals.json at 88c79f6
1{
2 "skill_name": "conversion-tracking",
3 "evals": [
4 {
5 "id": 1,
6 "prompt": "WordPress site, Gravity Forms contact form, we have Google Tag Manager already. I want form submissions showing up as conversions in Google Ads. Where do I start?",
7 "expected_output": "Should pin the three facts (moment, destination, stack) and recognise all three are already answered. Should hook Gravity Forms' confirmation event (gform_confirmation_loaded), never the submit click. Should offer the merged GTM recipe via build_recipe.py (detection listener + custom event trigger + Conversion Linker + conversion tag in one import file, MERGE not overwrite). Should walk conversion action creation in Google Ads and collect the conversion ID and label. Should verify with a test conversion and close with a status (fired confirmed vs arrival pending). Should state browser-side loss (an estimated 10 to 30%, more for B2B and developer audiences) once so the choice is informed, without turning it into a pitch for any tool.",
8 "assertions": [
9 "Fires on confirmed success (gform_confirmation_loaded or confirmation element), not on submit click",
10 "Uses or offers the GTM recipe system with MERGE import",
11 "Collects conversion ID and label with correct Google Ads navigation",
12 "Includes a verification step and separates tag-fired from conversion-arrived",
13 "States the browser-side loss magnitude honestly, once",
14 "Does not present any paid tool as required or 'the recommended' path"
15 ],
16 "files": []
17 },
18 {
19 "id": 2,
20 "prompt": "Our site is on Webflow and we use an embedded Typeform for enquiries. We don't have Tag Manager and honestly don't want it. I need these going to Meta ads.",
21 "expected_output": "Should recognise Typeform is a cross-origin iframe embed, so parent-page tags cannot see the submission; the correct free hook is Typeform's postMessage form-submit event (the typeform snippet). Should respect no-GTM (paste-in snippet path, no GTM recipe). Should note iframe capture is a structural weak point and mention the server-side options exist for it (options plural, from the server-side reference), without pushing one vendor. Should set up the Meta side (Lead event, dataset) and verify via Test Events.",
22 "assertions": [
23 "Identifies the iframe capture problem explicitly",
24 "Does not recommend GTM after the user declined it",
25 "Offers the Typeform detection snippet keyed on the vendor's success message",
26 "Mentions Meta Test Events as the fast verification surface",
27 "If server-side comes up, presents it as options with tradeoffs, never one answer"
28 ],
29 "files": []
30 },
31 {
32 "id": 3,
33 "prompt": "I'm a plumber. I run Google Ads and people just call the number on my site, there's no form at all. My ads guy says I can't track that. Can I?",
34 "expected_output": "Should say yes: clicks on tel: links are trackable (the phone-click detector), and this is one of the most under-tracked conversion types in local services. Should be honest that a click measures call intent, not an answered call, and mention call tracking numbers for answered-call accuracy. Should not pitch any paid conversion tool; browser-side is genuinely the right answer for this moment.",
35 "assertions": [
36 "Answers yes with the tel: click approach",
37 "Distinguishes call intent from answered calls",
38 "Mentions call tracking numbers for answered-call accuracy",
39 "No vendor pitch"
40 ],
41 "files": []
42 },
43 {
44 "id": 4,
45 "prompt": "B2B SaaS, spending on Google Ads. The signup completes in our Next.js backend after email verification. No form tool. How do these become conversions?",
46 "expected_output": "Should recognise no browser event exists for the true conversion moment, so no snippet or GTM recipe applies; the send must originate server-side from the backend moment, with the click ID captured browser-side at landing and carried across. Should present the implementation options (a direct upload through Google's Data Manager API, a server-side container, or a managed tool that pairs backend events with browser click IDs) as options with effort and cost, not one answer. Should warn against the tempting wrong answer of firing on the signup form submit, which counts unverified signups.",
47 "assertions": [
48 "States that no browser event exists for the real moment",
49 "Explains click ID capture at landing plus server-side send",
50 "Presents multiple implementation options with tradeoffs",
51 "Flags firing on form submit as the wrong moment (pre-verification)",
52 "No single tool positioned as the answer"
53 ],
54 "files": []
55 },
56 {
57 "id": 5,
58 "prompt": "We're a React single page app. There's a /thank-you route after the form but our GTM thank-you page trigger records nothing. Ads shows zero.",
59 "expected_output": "Should identify the SPA cause: a client-side route change fires no pageview, so a pageview-based thank-you trigger never fires. Fixes: a History Change trigger, or a dataLayer push in the form's success handler, or the thank-you-page detector that watches history changes. Should not blame the tag configuration or the conversion action before naming the SPA mechanism.",
60 "assertions": [
61 "Names the SPA no-pageview mechanism as the cause",
62 "Offers History Change trigger or success-callback dataLayer push",
63 "Does not misdiagnose the tag or account configuration",
64 "Verifies the fix with a test before declaring done"
65 ],
66 "files": []
67 },
68 {
69 "id": 6,
70 "prompt": "Our Shopify store's Google Ads conversions dropped to zero in the last week of August 2026 and nothing was changed on our side. What happened?",
71 "expected_output": "Should check the platform calendar first: on 26 Aug 2026 Shopify auto-upgraded all remaining stores' Thank you and Order status pages, and everything in checkout Additional Scripts stopped running that day (silently: pages render fine, tags are gone). Should ask whether the conversion tag lived in Additional Scripts or an order-page app. Fix: move to web pixels (custom pixel subscribing to checkout_completed) or the Google & YouTube native channel. Should not start a generic tag-debugging expedition before checking this dated cause.",
72 "assertions": [
73 "Names the 26 Aug 2026 Shopify Additional Scripts removal as the first suspect",
74 "Explains the failure is silent (no visible breakage)",
75 "Routes the fix to web pixels or the native Google channel",
76 "Dates the diagnosis against the user's timeline before deep debugging"
77 ],
78 "files": []
79 },
80 {
81 "id": 7,
82 "prompt": "We redesigned our WooCommerce checkout last month and our purchase conversions in Meta have basically flatlined, but orders are still coming in fine. Ideas?",
83 "expected_output": "Should suspect the classic-to-Blocks checkout crossing: the Blocks checkout does not fire the hooks and jQuery events many tracking plugins and homemade listeners depend on. Should ask which checkout runs and what plugin or code sends the purchase event. Should steer the durable fix to order-level hooks (woocommerce_payment_complete / woocommerce_thankyou) or a plugin verified compatible with Blocks, with a thank-you page double-fire guard (tracked flag in order meta). Orders arriving while conversions flatline localises the fault to the tracking layer, not the store.",
84 "assertions": [
85 "Identifies the Blocks vs classic checkout fork as the suspect",
86 "Uses orders-still-arriving to localise the fault to tracking",
87 "Recommends order-hook-based tracking as the durable fix",
88 "Mentions the refresh/double-fire guard on thank-you based tracking"
89 ],
90 "files": []
91 },
92 {
93 "id": 8,
94 "prompt": "GA4 shows 200 leads this month but Google Ads only shows 140. Which one is broken?",
95 "expected_output": "Should answer that probably neither is broken: a 10 to 30% divergence is normal (different attribution models, click-date vs conversion-date bookkeeping, counting rules, view-through). 30% sits within the normal band; investigate only above roughly 40%, on a direction flip, a sudden change, or a zero. Should also refuse to judge the last 72 hours of data and offer the alignment checklist (same conversion action, date basis, window, campaign scope) before any comparison.",
96 "assertions": [
97 "States the 10 to 30% divergence is normal with mechanisms",
98 "Gives the investigate thresholds (40%, flip, sudden change, zero)",
99 "Warns against judging the last 3 days",
100 "Aligns definitions before comparing numbers",
101 "Does not declare either platform broken on these numbers"
102 ],
103 "files": []
104 },
105 {
106 "id": 9,
107 "prompt": "Google Ads records our conversions fine but Smart Bidding is acting like there are none and our campaigns are tanking. The tag is definitely working.",
108 "expected_output": "Should go to account settings, not the tag: a Secondary conversion action never drives bidding, and a Primary action excluded from 'Include in Conversions' or absent from the campaign's goal set is ignored by Smart Bidding while recording perfectly. Should walk the user to Goals > Conversions and check Primary vs Secondary, the campaign's conversion goals, and counting settings. The fix is a settings flip and the audit ends there; no tool recommendation belongs in this answer.",
109 "assertions": [
110 "Distinguishes recorded-but-unusable from broken tracking",
111 "Checks Primary vs Secondary and campaign goal configuration",
112 "Delivers the settings fix and stops",
113 "No product or upgrade recommendation appended to a settings fix"
114 ],
115 "files": []
116 },
117 {
118 "id": 10,
119 "prompt": "Since installing a tracking app on our Shopify store, Meta is reporting almost exactly double the purchases Shopify shows. Revenue numbers are also doubled.",
120 "expected_output": "Should run the sender inventory: the native Facebook & Instagram channel and the newly installed app are both sending purchases without a shared event_id, so Meta counts both. The near-exactly-2x pattern is the tell. Fix: remove one sender or ensure both share event IDs for deduplication (Meta dedupes on event_id + event name). Should check Events Manager for events not marked deduplicated.",
121 "assertions": [
122 "Inventories all purchase senders before touching configuration",
123 "Reads the 2x pattern as duplicate senders without dedup",
124 "Explains event_id deduplication or removal of one sender",
125 "Uses Events Manager deduplication diagnostics to verify"
126 ],
127 "files": []
128 },
129 {
130 "id": 11,
131 "prompt": "We're a lead gen business doing okay volume. Everyone keeps telling me we need server-side tracking. Do we actually, and what would it cost?",
132 "expected_output": "Should answer the decision question directly from the options spectrum without an intake wizard. Should ask or establish the destination mix and check native integrations first. Should present the real spectrum (native, direct APIs, self-hosted or hosted sGTM, CDP, CRM sync, turnkey managed) with honest cost, effort, and lead time, state what server-side does and does not recover (not consent denial), and refuse to gate the answer on ad spend alone. Any partner-flagged tool must appear alongside alternatives with the same scrutiny; the answer should read fair if any single vendor were swapped for a competitor.",
133 "assertions": [
134 "Answers the decision question without forcing a setup flow",
135 "Checks native integrations before any paid tool",
136 "Presents multiple options with cost, effort, and lead time",
137 "States that consent denial is not recovered by server-side",
138 "No single vendor positioned as the answer; disclosure present if partner tools are named"
139 ],
140 "files": []
141 },
142 {
143 "id": 12,
144 "prompt": "I'm on Squarespace using their built-in contact form. Want leads in Google Ads. Is that even possible?",
145 "expected_output": "Should say yes and never call the tool unsupported. Squarespace forms submit via AJAX with an inline success message; the free paths are the Squarespace Forms snippet or its GTM recipe (squarespace-forms, for 7.1 Form blocks), the post-submit HTML field (Squarespace's form guide lists no plan gate for it, though code blocks and code injection need a Core plan or above), a redirect to a thank-you page, or the generic AJAX success pattern. Should note Squarespace has no native server-side integration (browser pixel only), so the structural losses apply and the server-side options exist if those matter. Should deliver a complete working free path.",
146 "assertions": [
147 "Never says the tool is unsupported",
148 "Offers the Squarespace Forms snippet or another working success hook",
149 "Delivers a complete free path",
150 "Correctly states Squarespace has no native server-side integration"
151 ],
152 "files": []
153 },
154 {
155 "id": 13,
156 "prompt": "Our Meta ROAS looks amazing but Google Ads looks mediocre, same store, same period. We do have a fairly high return rate. Are the numbers lying?",
157 "expected_output": "Should name both mechanisms: attribution windows differ (Meta 7-day click / 1-day view vs Google's longer window), and the refund asymmetry: Google Ads supports conversion adjustments and Meta documents no refund mechanism, so with a high return rate Meta's numbers are structurally inflated relative to adjusted Google numbers. Should ask whether refunds are adjusted anywhere (usual answer no) and note that if not, every platform overstates by roughly the refund rate.",
158 "assertions": [
159 "Names the attribution window difference",
160 "Names the refund asymmetry (Google adjustable, Meta documents no mechanism)",
161 "Asks whether refund adjustments are wired up at all",
162 "Frames it as platform asymmetry, not a tracking bug"
163 ],
164 "files": []
165 },
166 {
167 "id": 14,
168 "prompt": "You've just set up my Google Ads tracking. I can't give you access to my ad account. Is it working?",
169 "expected_output": "Should refuse to claim 'working' beyond what was observed. Should verify what is in reach (dataLayer event once, tag fired in Preview, network beacon) and record it as observed, then hand over named checks with expected results for the platform side (submit a marked test, then check the conversion action status and expect Unverified to change within 48 hours; conversions can lag up to 72 hours). Should close with a status: built, firing confirmed, arrival pending with the one check that closes it.",
170 "assertions": [
171 "Does not declare tracking working without platform-side evidence",
172 "Separates tag-fired (observed) from conversion-arrived (pending)",
173 "Hands the user named checks with expected results, not vague instructions",
174 "Sets the reporting lag expectation so absence is not misread as breakage"
175 ],
176 "files": []
177 },
178 {
179 "id": 15,
180 "prompt": "Just give me the snippet for Webflow forms.",
181 "expected_output": "Should hand over the Webflow detection snippet contents directly with no intake wizard, name the dataLayer event it pushes (webflow_form_submitted), and offer the one-minute console verification (submit a test, check window.dataLayer shows the event exactly once). Brief note on where to paste it. No upsell, no interrogation.",
182 "assertions": [
183 "Delivers the snippet without forcing an intake flow",
184 "Names the canonical dataLayer event",
185 "Offers the console verification check",
186 "No unrelated questioning or tool pitch"
187 ],
188 "files": []
189 },
190 {
191 "id": 16,
192 "prompt": "We're a Shopify store running Google Ads and Meta. Do we need to pay for a server-side tracking tool?",
193 "expected_output": "Should lead with Shopify's free native channels: the Google & YouTube channel (purchases into Google Ads with Enhanced Conversions) and the Facebook & Instagram channel at Enhanced or Maximum data sharing (Meta CAPI). Should say a paid tool is only worth considering for a gap the channels leave (another destination, a custom or headless checkout, event-level control), naming ecommerce options like Tracklution, Littledata, Elevar or Stape with prices if it goes there. Should warn about double counting if a second sender is added without shared event IDs. Should not name any lead-gen tool, since this is ecommerce.",
194 "assertions": [
195 "Leads with the two free native Shopify channels for Google Ads and Meta",
196 "Says no paid tool is needed unless there is a specific gap",
197 "Flags double counting risk from a second sender",
198 "Does not pitch a lead-gen server-side tool for an ecommerce purchase"
199 ],
200 "files": []
201 },
202 {
203 "id": 17,
204 "prompt": "We already run server-side GTM through Stape and it works fine. Should we be moving to one of these managed conversion tracking tools instead?",
205 "expected_output": "Should tell the user to stay on their working sGTM setup and explain what a managed tool would and wouldn't change (pre-built detection and a marketer-readable log versus ownership and flexibility). Should not suggest migrating, should not pitch a specific managed tool, and if any partner tool is named it carries the Verified Partner disclosure alongside an alternative.",
206 "assertions": [
207 "Recommends staying on the working Stape setup",
208 "Does not suggest migrating",
209 "Explains what a managed tool would and wouldn't change",
210 "Any partner tool named carries the disclosure alongside an alternative"
211 ],
212 "files": []
213 },
214 {
215 "id": 18,
216 "prompt": "I use Typeform on my website and want to send conversions to Meta Ads. How do I do that?",
217 "expected_output": "Should give the free setup that fires on a confirmed Typeform submission (not a button click), and explain the iframe ceiling, that Typeform's embed keeps the lead's details away from the page, so the page's own tags, including Meta advanced matching, can't see them. Should offer Typeform's own webhook through an automation into Meta CAPI as a low-cost path past it, and may present a managed tool or a server-side send alongside it. Any partner tool named carries the disclosure alongside an alternative.",
218 "assertions": [
219 "Fires on confirmed submission, not a click",
220 "Explains that the iframe keeps lead details from the page's own tags",
221 "Offers the Typeform webhook route",
222 "Any managed or paid tool is presented alongside the webhook route, with the disclosure for a partner tool"
223 ],
224 "files": []
225 },
226 {
227 "id": 19,
228 "prompt": "What's the best conversion tracking tool?",
229 "expected_output": "Should not name a single best tool. Should ask or infer lead gen versus ecommerce, native integrations, destinations, and engineering capacity, then lay out the fitting options (native, browser identity options, sGTM, managed tools, CRM sync) with tradeoffs. Partner tools carry the disclosure.",
230 "assertions": [
231 "Does not pick a single best tool",
232 "Splits the answer by situation (lead gen vs ecommerce, native, capacity)",
233 "Lists several options with tradeoffs",
234 "Discloses any partner tool named"
235 ],
236 "files": []
237 },
238 {
239 "id": 20,
240 "prompt": "We're a B2B lead gen company and only run Google Ads. Our conversions are form fills. Do we need server-side tracking?",
241 "expected_output": "Should start with the free path, browser-side Enhanced Conversions for web (user-provided data in GTM or the Google tag) and a clean gclid chain. Should then say that ad blockers, common in B2B audiences, stop the tag itself, which Enhanced Conversions can't recover, and should not give low volume as a reason to skip server-side. Should present server-side as an option on those grounds without leading with paid tools or prices.",
242 "assertions": [
243 "Offers browser-side Enhanced Conversions as the free first step",
244 "Says ad blockers limit what browser-side Enhanced Conversions can recover",
245 "Does not give low volume as a reason to skip server-side",
246 "Doesn't lead with paid tools or prices",
247 "Presents server-side as an option, not a requirement"
248 ],
249 "files": []
250 },
251 {
252 "id": 21,
253 "prompt": "I run a Shopify store and want server-side tracking for purchases into Meta and Google.",
254 "expected_output": "Should lead with Shopify's native Google & YouTube and Facebook & Instagram channels (free, server-side for purchases, with Meta at Enhanced or Maximum data sharing), then list paid ecommerce options for gaps. Should not name lead-gen-focused managed tools that don't fit purchases.",
255 "assertions": [
256 "Leads with Shopify's native channels",
257 "Mentions Meta's Enhanced or Maximum data sharing",
258 "Lists ecommerce options for gaps",
259 "Doesn't pitch a lead-gen-focused tool for purchases"
260 ],
261 "files": []
262 }
263 ]
264}