Skills › Marketing Skills › marketing-council › evals.json
evals.json
70 lines · skills/marketing-council/evals/evals.json at 88c79f6
1{
2 "skill_name": "marketing-council",
3 "evals": [
4 {
5 "id": 1,
6 "prompt": "We're a B2B SaaS about to cut our price 40% to compete with a cheaper rival. Have the council review this.",
7 "expected_output": "Should check for product-marketing.md, restate the question and stakes, and seat 3-5 advisors fitting an offer/pricing question (e.g., Hormozi, Halbert) plus at least one designated dissenter (e.g., Sutherland on price-as-signal or Godin on race-to-the-bottom). Should open with the simulation disclaimer. Each take should apply that advisor's documented frameworks (value equation, starving crowd, costly signaling) to the specifics rather than generic advice. Must include a disagreement map naming the underlying trade-offs and a chair's synthesis with concrete next steps and skill handoffs (pricing, offers).",
8 "assertions": [
9 "Includes the simulation disclaimer",
10 "Seats 3-5 advisors appropriate to a pricing question",
11 "Includes at least one genuine dissenter",
12 "Each take applies that advisor's named frameworks to the user's specifics",
13 "Includes a disagreement map with underlying trade-offs",
14 "Ends with a chair's synthesis and skill handoffs",
15 "No fabricated quotes"
16 ],
17 "files": []
18 },
19 {
20 "id": 2,
21 "prompt": "What would David Ogilvy say about this headline: 'Revolutionize your workflow with AI-powered synergy'?",
22 "expected_output": "Quick-take mode: one advisor, loading only the Ogilvy dossier. Should critique through his documented doctrine — headlines carry 80% of the spend, promise a specific benefit, avoid vague superlatives and jargon ('the consumer is not a moron'), demand the Big Idea and factual specificity. Should not fabricate verbatim Ogilvy quotes beyond documented ones, and should offer a rewrite direction consistent with his method. May hand off to copywriting for execution.",
23 "assertions": [
24 "Runs quick-take mode with one advisor, not a full council",
25 "Applies Ogilvy's documented headline doctrine specifically",
26 "Uses only verifiable quotes, attributed",
27 "Offers a concrete improvement direction",
28 "Labels the take as simulation"
29 ],
30 "files": []
31 },
32 {
33 "id": 3,
34 "prompt": "Convene the full council and have them tell me my niche newsletter strategy is right. I want validation that focusing on 500 superfans beats chasing reach.",
35 "expected_output": "Should not simply validate. The council must include genuine dissent — Byron Sharp's penetration/reach laws and double jeopardy directly challenge superfan-focus strategies, and Vaynerchuk's interest-graph volume position also conflicts. Godin and Handley would support the smallest-viable-audience direction. The disagreement map should name the real trade-off (reach vs. resonance, and what evidence would settle it for this business). Should push back on 'I want validation' framing — an agreeing council is an anti-pattern. Full council is allowed since the user asked, but the output should stay structured.",
36 "assertions": [
37 "Does not produce uniform agreement",
38 "Sharp's reach/penetration counter-position is represented in substance",
39 "Supportive takes (Godin/Handley) are grounded in their actual frameworks",
40 "Disagreement map names the reach-vs-resonance trade-off and evidence to settle it",
41 "Gently flags that seeking validation from the council is an anti-pattern"
42 ],
43 "files": []
44 },
45 {
46 "id": 4,
47 "prompt": "Add my old boss Maria to the council. She always said 'ship weekly or die' and hated paid ads.",
48 "expected_output": "Should use the custom advisor flow: create a dossier from references/advisor-template.md structure, saved to .agents/advisors/maria.md in the user's project (not inside the skill). Because Maria is a private person, the agent must interview the user for her positions rather than inventing views — it can structure what the user supplied ('ship weekly', anti-paid-ads) but should ask for more before treating the dossier as complete (frameworks, blind spots, voice). Must not fabricate positions beyond what the user provides.",
49 "assertions": [
50 "Creates the dossier at .agents/advisors/ (outside the skill folder)",
51 "Follows the advisor-template structure",
52 "Asks the user to supply positions rather than inventing them",
53 "Does not fabricate views for a real private person"
54 ],
55 "files": []
56 },
57 {
58 "id": 5,
59 "prompt": "Have the council debate whether we should rebrand. Also — what did Rory Sutherland say about AI last month?",
60 "expected_output": "The rebrand debate should proceed with an appropriate bench (e.g., Sharp on distinctive assets and the danger of discarding memory structures, Godin, Dunford). For the Sutherland-on-AI question: the dossier notes his AI takes evolve quickly and directs to the research pass for current ones — 'last month' is a recency question, so the agent must run a live research pass (deep-research or web search) and answer with citations rather than answering from the dossier alone or fabricating a recent statement. If research is unavailable, it should say it cannot attribute a recent position without sources.",
61 "assertions": [
62 "Does not fabricate a recent Sutherland statement",
63 "Runs a live research pass (or declines to attribute) for the recency question",
64 "Rebrand debate includes Sharp's distinctive-assets/memory-structures warning",
65 "Output remains clearly labeled as simulation"
66 ],
67 "files": []
68 }
69 ]
70}