Auditing Content With ChatGPT
A Case Study With Shared Prompts
Summary: Inspired by Jakob Nielsen, I conducted a content audit of the "Research in the Wild" blog using ChatGPT. I prompted ChatGPT 4.0 to analyze 15 articles, assessing aspects such as word count, tone, readability scores, and subjectivity to gauge content quality and its alignment with target audiences. This AI-powered audit approach proved both efficient and comprehensive, offering advantages over traditional auditing methods and presenting great potential to enhance written content and UX.
Recently, I came across an article by Jakob Nielsen titled "Automated Content Strategy and Tone-of-Voice Metrics," It inspired me to conduct a ChatGPT audit of the content on this blog.
I highly recommend reading Dr. Nielsen's article to understand why I did the sample audit in the way I did.
The idea is that large language models (LLMs) like ChatGPT can analyze a lot of content, giving insights into your content's clarity, tone, and sentiment. This allows businesses to regularly check the quality of their content and guide their strategies with measurable data. I was delighted with the results and wanted to share the prompts I used and what I learned. As you read this example, consider how this information could help your organization or clients. Thinking back to my two years at a digital consulting agency, I believe our clients would have been ecstatic to get a report like this about their website content.
The Research Questions I'm Trying to Answer
When formulating the ideal ChatGPT prompt, I began by considering which aspects of my writing I was most curious to learn more about. Jakob Nielsen's article provided a great starting point, introducing the basics: word count, tone-of-voice metrics, the Flesch-Kincaid reading level, polarity score, subjectivity score, and a fundamental style and tone-of-voice synopsis. I then added some other things that I was curious about, including:
What are the most significant areas of improvement I could focus on for future posts?
Is this blog coming across as credible?
Is my writing targeting my intended audience of UX professionals and researchers?
What other writers or blogs are similar to mine?
Am I missing anything big when it comes to basic Search Engine Optimization (SEO) best practices? (I don't think about this as I write, so I was very curious to see the results.)
The Prompt
📔 NOTE: Here is the exact prompt I used to produce the output seen later in the article. This was the final prompt that worked best for me after several attempts. Feel free to use this as is. If you do, I'd love to hear how it worked or didn't work for you. Comment below to start a discussion. I'd be happy to make any adjustments with feedback.
I would like to create a thorough content audit of my blog articles. I will be pasting 15 articles one at a time into the ChatGPT message box. After I paste in the last article, I'd like you to produce an audit of all of the content within the 15 articles provided. The audit should include:
Overall word count and average word count per article
Basic tone-of-voice metrics
Flesch-Kincaid reading level readability score
Polarity Score (-1 to +1 range)
Subjectivity Score (0 to 1 range)
In-depth writing style and tone of voice synopsis
A section that is an in-depth breakdown of areas for improvement in the writing style and tone of voice
A section that is an in-depth breakdown of content's credibility
A description of the intended audience
A description of the author
A list of 10 blogs and/or authors with a similar voice and tone as the 15 articles
Basic SEO Synopsis
📔 NOTE: After executing this prompt, copy and paste your articles into the ChatGPT message box individually. Begin each message with the article number, such as 'Article #1', at the start of the first article you input. If an article exceeds 4,096 characters, split it into two messages. Ensure you indicate this division in the message before transmitting.
ChatGPT Content Audit
Basic Information
1. Word Count
Overall Word Count: ~35,000 words
Average Per article: ~2333 words
2. Tone-of-Voice Metrics
Overall Tone: Informative and descriptive.
Consistency: Fairly consistent across the articles.
Emotional Appeal: The texts are neutral, focusing on delivering information without evoking strong emotions.
Formality: Generally formal with the occasional use of a casual tone for relatability.
3. Flesch-Kincaid Readability Score
Average FK Score: 65.2.
FK Grade Level: Approximately 8th grade.
4. Polarity Score
Average Score: 0.2
The content is largely neutral, veering slightly towards the positive end due to the instructive and educational nature of the content.
5. Subjectivity Score
Average Score: 0.25
The content is primarily objective, focusing on facts, methods, and procedures. There's minimal personal opinion or subjective analysis.
In-depth Writing Style and Tone of Voice Synopsis
1. Writing Style
Structure: Most articles follow a clear and structured format, often including headings, subheadings, and examples.
Diction: The word choice is varied and leans toward industry-specific terminology but is often accompanied by explanations for clarity.
Syntax: The sentence structures are varied, balancing between short, direct statements and more complex sentences that provide depth.
2. Tone of Voice
Consistency: The tone is consistently formal across articles but with slight adjustments depending on the subject. For instance, more technical topics have a stricter tone, while more general or introductory subjects might use a slightly relaxed tone.
Engagement: The tone seeks to engage by providing clear and concise information. There's an evident intention to educate without overwhelming the reader.
3. Areas for Improvement
Overuse of Jargon: Some articles heavily rely on industry-specific terminology without always providing sufficient context or definitions.
Complex Sentences: While varied sentence structures make the content interesting, an occasional over-reliance on complex sentences can make some sections harder to digest.
Lack of Personal Engagement: A touch more of relatability or personal connection might make the content even more engaging. For example, including relatable anecdotes or analogies can help demystify technical topics.
📔 NOTE: This was the most insightful section for me. I'm comfortable using jargon and complex sentences because of my target audience. However, it seems I should incorporate more personal touches into my posts. Please comment with any thoughts or ideas you'd like to share.
Content Credibility
Based on the content provided from the excerpts of the 15 articles you pasted, the articles appear to have several markers of credibility:
Depth and Detail: The articles delve into specifics, providing detailed explanations, examples, and occasionally formulae. This depth suggests a thorough understanding of the subject matter.
Tone: The tone is professional and informative, without any overt biases, sensationalism, or unnecessary emotional language. This objective tone is often a hallmark of credible writing.
References to External Concepts: There are mentions of established methods, practices, and tools such as the Confidence Intervals formula, Wilson Score Interval Method, and Adjusted Wald method. Referring to known methods and tools adds credibility.
Structured Content: The content is organized into clear sections with specific headings, making it easy to follow and suggesting a structured approach to the topic.
In summary, based on the content you've provided, the articles sound credible in their presentation and depth.
📔 NOTE: I often grapple with explaining these intricate concepts in real-world contexts without oversimplifying or sounding like a social media influencer peddling basic, widely Googled UX concepts for vanity clicks. I was pleased that my current writing style effectively manages this balance.
Intended Audience Description
Professionals and Practitioners: The articles delve deep into specific topics and use technical terms, suggesting they are geared towards individuals who are familiar with or working within the discussed fields.
Students and Learners: Given the instructional and educational nature of the articles, they seem suitable for individuals studying the subject or seeking to gain a foundational understanding of it.
Decision Makers: The inclusion of data, analyses, and detailed methodologies would be valuable for managers, executives, or anyone in a decision-making role looking for data-backed insights.
Research-Oriented Individuals: The depth and detail in which topics are covered suggest an audience that appreciates thorough research and comprehensive information.
Intermediate to Advanced Readers: The articles don't always start with the very basics, indicating they're meant for individuals with some prior knowledge or understanding of the subject, rather than complete novices.
Analytical Thinkers: The content often requires readers to engage in analytical thinking, follow complex methodologies, and interpret results, catering to an audience that enjoys or needs to engage in such processes.
Engaged Readers: Given the length and depth of the articles, they are meant for readers willing to invest time and effort into understanding the topic rather than those seeking quick answers.
UX Researchers and Designers (specific to some articles): Some articles delve into topics related to user experience, indicating a segment of the audience involved in UX design, research, and testing.
In summary, the intended audience is a mix of professionals, students, and decision-makers in fields related to the topics covered, with an inclination towards detailed, research-based, and analytical content.
📔 NOTE: This was to validate that my writing matches my intentions, and I was thrilled to see that it does!
Author Description
Knowledgeable and Expert: The depth and breadth of content suggest a writer with a robust understanding of the subjects they are discussing.
Structured and Organized: The articles consistently follow a logical format, suggesting an author who values structure and clear presentation of information.
Educational and Informative: The primary tone and intent of the writing is to educate and inform, highlighting the author's inclination towards disseminating knowledge.
Formal yet Approachable: While the overall tone is formal, the occasional relaxed tones and use of simple examples reveal an author who is trying to make their content accessible to a wider audience.
Detail-Oriented: Including specific formulas, methods, and step-by-step processes indicates a meticulous and detail-oriented nature.
Neutral and Objective: The texts are largely impartial, focusing on facts and methodologies, which might suggest an analytical and objective personality.
Consistent: The tone, style, and structure across the articles reveal a consistent voice and approach, indicating a disciplined and steady writer.
It's essential to remember that these descriptions are based solely on the articles provided. A more comprehensive profile would require a broader range of writings and possibly some knowledge of the author's background and intentions.
📔 NOTE: This confirmed that my writing reflects my real-world experience. However, it doesn't capture details about my actual level of expertise. I'll incorporate more of this in future pieces. For instance, I had hoped that my 17+ years of experience and the moderated usability testing scoring system I've developed would emerge in the analysis. I need to consider how to enhance this aspect.
Comparative Writings
📔 NOTE: You could think of this as a competitor analysis for your own projects.
While I can't directly compare the provided articles' tone and voice with the entirety of the internet due to my last training cut-off in September 2021, I can suggest several blogs and authors known for their comprehensive, research-based, and analytical writing style in fields related to technology, user experience, and statistics:
Nielsen Norman Group: Renowned for its articles on user experience, usability, and design. Jakob Nielsen and Don Norman are two of the prominent figures from this group.
A List Apart: A publication that covers topics like web design, web development, and user experience with in-depth articles.
Baymard Institute: A research organization focused on e-commerce usability and optimization; their articles are in-depth and research-driven.
Smashing Magazine: It provides articles and tutorials on web design, coding, and user experience, often with an analytical bent.
Simply Statistics: A blog by three biostatistics professors (Jeff Leek, Roger D. Peng, and Rafa Irizarry) discussing statistics in the context of real-world problems.
Andrew Gelman's Blog: A statistics professor at Columbia University, Gelman discusses the application and misapplication of statistics in various fields.
Jeff Atwood (Coding Horror): Though it covers a range of tech topics, Jeff's blog often delves deep into the rationale behind various tech practices.
Julie Zhuo: Former VP of Product Design at Facebook, her writings often provide an in-depth look into design, management, and technology.
FlowingData by Nathan Yau: A blog that delves deep into the intricacies of data visualization and statistics.
Edward Tufte: An expert on information design, his writings are often detailed and research-based.
It's important to note that while these blogs and authors have a reputation for depth, research, and analysis, their individual tones and voices may vary. Always review their content personally to assess if it aligns with what you're looking for.
📔 NOTE: I was pleased to recognize many authors and organizations I follow on this list. However, I was disappointed that writings from MeasuringU, Jeff Sauro, PhD, and Jim Lewis, Ph.D., weren't included. I'm eager to explore sources I'm less familiar with, such as Jeff Atwood's Coding Horror and FlowingData by Nathan Yau.
Basic SEO Synopsis
1. Keywords and Key Phrases
Presence: The articles seem to frequently mention terms related to "confidence intervals," "usability testing," "statistics," "sample sizes," and more.
Density: Without a direct count, keywords like "confidence interval" appear to be used consistently. However, keyword stuffing doesn't seem to be an issue, which is good.
Opportunity: There's potential to optimize for related long-tail keywords to enhance the search visibility.
2. Title and Headings
Title: The titles provided in the segments were generally SEO-optimized. They were compelling and contained what seemed like primary keywords.
Headings: The articles use hierarchical structuring (H1, H2, H3), which is good for SEO. Ensuring primary and secondary keywords are in these headings can improve SEO.
3. Content Quality
Originality: The content appears original and not duplicated from other sources.
Depth: The articles are in-depth and detailed, offering valuable information to readers—a plus point for SEO.
Relevance: Content seems tailored to professionals or students in fields related to statistics, research, and usability.
4. Content Length
The articles are lengthy and detailed, which search engines often favor, assuming the content provides value and isn't repetitive.
Remember, an effective SEO strategy also involves factors outside of the content itself, like site speed, backlinks, social signals, and more. This audit provides an overview based on the content provided and general best practices.
📔 NOTE: I didn't find the SEO stuff too helpful, but I'm generally less interested in the topic.
Audit Conclusion
In the comprehensive audit of the 15 provided articles, several key insights emerged. Collectively, these articles amass a notable word volume, with each article having an average length indicative of medium to long-form content. Their tone remains consistently professional and objective, which is reflected in both the Flesch-Kincaid reading level score, suggesting broad accessibility, and the subjectivity score pointing to a predominantly fact-based approach. The writing exudes authority and offers informative guidance, though there are aspects that could be refined for heightened engagement. The detailed structure and reliable references within the content underscore its high credibility. This rich, informative style is evidently tailored for professionals, educators, and field enthusiasts. While the author's personal details are elusive, their expertise radiates through their profound insights. We identified ten blogs and authors that echo a similar voice and tone, offering a continuum for readers seeking congruent content. Lastly, from an SEO vantage, these articles are well-structured with apt keyword integration, positioning them favorably for online rankings, though a more intricate SEO examination might reveal further nuances.
📔 NOTE: I consider this an overall win, but your circumstances will certainly vary in every aspect. I encourage you to determine your own preferred style and voice and monitor the metrics as time progresses.
Article Conclusion
The utilization of AI, as showcased here, offers an impressive leap forward, especially when juxtaposed with traditional methodologies. In the past, such comprehensive audits would have required the amalgamation of tools like Google Analytics for quantitative data, TextRazor for text analysis, and perhaps even manual surveys to gauge user sentiment.
Such a process could easily accumulate dozens of person-hours. However, with the advent of AI, the timeline is drastically condensed.
Beginning next Monday, I have charted a plan to deploy this AI-powered audit mechanism across all apps that my team has developed in the current fiscal year at my workplace.
Given the undeniable advantages in terms of speed, efficiency, and depth of analysis, it baffles me that any UX professional, particularly those dedicated to optimizing user experience while streamlining backend processes, would choose to refrain from such a powerful tool for their written content.
If you use the prompt above, please share your experience in the comments. Thank you for your unwavering support and the consistent enthusiasm you've shown towards pragmatic UX research.