How to Find Out If Someone Used ChatGPT
Artificial Intelligence

How to Find Out If Someone Used ChatGPT

Mandi D 19 November 2023

Since its public debut in November 2022, ChatGPT has captured global attention for its ability to generate human-like text. The AI chatbot created by OpenAI can craft remarkably articulate and coherent responses to prompts, displaying conversational skills that were unimaginable just years ago.

However, ChatGPT’s advanced capacities also present new dilemmas. As text generation reaches frighteningly sophisticated levels, how can we discern machine-created content from human originality? In spheres like academia and journalism where integrity and truth matter most, ChatGPT threatens to enable plagiarism and misinformation if deployed irresponsibly.

Detecting ChatGPT’s involvement has thus become an urgent priority. This article provides indispensable guidance for parsing human authorship from machine impersonation. By analyzing linguistic patterns, testing AI blindspots, and spotlighting logical gaps, you’ll gain tactics for picking out ChatGPT’s fingerprints.

Overview

This article provides comprehensive methods for detecting the use of ChatGPT and other AI writing assistants. By becoming familiar with their linguistic limitations and quirks, telltale grammatical errors, limited contexts, and susceptibility to prompting inconsistency, we can maintain authorial transparency and ethical AI use.

Key skills covered include:

  • Evaluating writing style, complexity, and coherence
  • Testing an excerpt through re-prompting ChatGPT itself
  • Assessing over-adherence to prompts and instructions
  • Spotting patterned errors and limitations
  • Comparing contexts and factual consistency

Read on to learn proven techniques for discerning human from AI writing across various contexts, from academic integrity to creative work and journalism. With vigilance and care, we can continue benefiting from AI assistants while upholding ethics.

Evaluating Writing Style and Complexity

One of the biggest giveaways that a piece of writing came from an AI like ChatGPT is its style and complexity – or lack thereof. While ChatGPT can produce human-sounding writing on a superficial level, a careful stylistic analysis will reveal key differences from authentic human writing:

Lack of Voice and Personality

  • Human writing has a recognizable voice, personality, perspective, and flair that reflects the author’s unique experiences and way of seeing the world. A keen human author leaves their imprint through word choice, phrasing selections, pop culture references, emphases, and opinions. There is a lively, idiosyncratic authorial presence.
  • In contrast, ChatGPT’s writing tends to sound generic, impersonal, and overly formal. The text lacks a compelling narrative voice and seems blandly disembodied. There is little trace of a distinct personality, background, values, or viewpoint. The tone is straightforward and clinical without deeper biases or framing shining through.

Formulaic and Predictable Structure

  • AI-generated text often follows predictable templates and structural conventions. For essays and articles, the introduction, thesis statement, topic sentences, transitional phrases, argument ordering, and conclusion tend to be formulaic and predictable.
  • Creative writing also adheres strongly to genre tropes and expected plot arcs. There is little deviation from established patterns and molds. Surprises and unpredictable structural choices are rare, giving the writing a manufactured and derivative quality on close inspection.

Lacks Subtlety and Nuance

  • ChatGPT struggles with nuanced takes, subtlety, and conveying connotations. The writing tends to be very on-the-nose, literal, and lacking in nuance. Figurative language, satire, irony, and deeper layered meanings are rare. Allusions and wordplay are unlikely to appear.
  • Instead, the AI writing remains at face value without hints of deeper subtext or reading between the lines. ChatGPT does not masterfully weave in subtle connections and connotations as a wordsmithing human can. Its takes come across as basic, blunt, and lacking in cleverness or hidden significance.

Weak Coherence and Flow

  • While reasonably coherent on a paragraph level, AI writing often falls apart in long-form coherence. The broader flow, organization, reasoning, and connectedness of ideas across an entire piece are weaker than human writing.
  • Logic gaps, unsupported connections, and choppy or illogical transitions become apparent when evaluating the writing holistically. ChatGPT fails to maintain tight integration and graceful progression across sections.

Light on Details and Examples

  • ChatGPT tends to skim over details, elaborate examples, vivid anecdotes, and concrete evidence. The writing stays high-level and abstract without properly illustrating or backing up points made.
  • This lack of convincing support is especially apparent in descriptive passages, scenes settings, and character details which come across as sparse and shallow. The AI struggles to conjure immersive sensory details compared to human creativity.

By keeping these nuanced style limitations in mind, you can more easily flag text that seems simplistic, formulaic, and lacking in nuance, voice, or creative flair. Of course, human writing can also exhibit some of these issues, so further careful analysis is required to definitively confirm AI use. But stylistic evaluation provides a strong starting point for detection.

Testing an Excerpt through Re-prompting

Re-prompting ChatGPT with a suspicious excerpt is one of the most definitive ways to evaluate if it was AI-generated. By analyzing how coherently andrelevantly ChatGPT continues the excerpt, you can gather strong evidence of original authorship.

When re-prompting, pay close attention to:

Continued Contextual Consistency

  • Does ChatGPT maintain logical flow from the excerpt, staying consistent with the established context, events, names, and facts? Or does it contradict established details, indicating it did not generate the original text?

Alignment of Writing Style

  • Does the newly generated text closely mirror the original excerpt in tone, complexity, voice, sentence structure, and word choices? High similarity suggests single authorship.

Presence of Redundancies

  • Does the new text repeat or rephrase information from the excerpt, indicating ChatGPT is redundantly covering the same context? This redundancy points to it originating the excerpt.

Relevance to Prompt

  • Does ChatGPT’s continued excerpt align with what you asked for, such as adding 2-3 coherent paragraphs? Or is the output off-topic, indicating it did not write the original.

Reaction to Alterations

  • Try slightly altering the excerpt before re-prompting, like changing a character name or plot point. Does ChatGPT detect and adapt to that change? Or does it ignore the alteration, signaling lack of authorship?

Run this re-prompting test from multiple angles by giving ChatGPT varied instructions such as continuing, summarizing, or critiquing the excerpt. The more the outputs align with the excerpt, exhibit redundancy, and reflect awareness of context, the stronger the indication that ChatGPT originally generated that text.

Checking accuracy of details against external sources can further validate authorship. Overall, strategically re-prompting excerpts with ChatGPT itself provides compelling evidence to make attribution judgments.

Assessing Over-Adherence to Prompts

ChatGPT frequently exhibits strong, almost rigid adherence to the initial prompts and instructions it is given when generating text. This over-literal interpretation and lack of deviation from the prompt can manifest in subtle but detectable ways when reviewing a passage that may have come from ChatGPT.

For instance, if a prompt asks ChatGPT to write a 5-paragraph essay on a certain topic, the resulting essay will almost certainly rigidly follow the exact 5-paragraph format laid out in the instructions, without any deviation or creative restructuring. The essay will contain precisely 5 paragraphs, even if another structure would have worked better to express the ideas.

In addition, the specific wording and phrases used in the initial prompt are often over-used and repeated verbatim throughout the AI-generated text. It’s as if ChatGPT fixates on prompt keywords and forces them in as much as possible rather than using more organic language.

Therefore, when reviewing a piece of writing that is suspected to have come from ChatGPT, look for signs of excessive prompt compliance and lack of deviation from the instructions. Does the writing strictly and possibly overly interpret and adhere to what the prompt requested?

For instance, if a prompt asks ChatGPT to “Use 5 examples to illustrate the theme,” the resulting text will probably laboriously provide those exact 5 examples without any additional ones that might have strengthened the point.

You can further validate this tendency by re-prompting ChatGPT with slightly altered requirements – such as asking for a 6 paragraph essay or 3 examples instead of 5. If ChatGPT generates something divergent from the original text based on these modified instructions, it provides strong confirming evidence that the original writing adhered very tightly to the original prompt down to the precise details.

Overall, ChatGPT’s lack of discretion and inability to deviate from prompts is a revealing autorship clue. While human writers interpret prompts more flexibly and creatively, ChatGPT’s programming forces strict compliance. Keeping an eye out for this adherence can help determine if an overly-prompt-faithful passage likely came from an AI.

Spotting Patterned Errors and Limitations

Despite improvements, ChatGPT still makes certain predictable errors reflective of its current training limitations. Being aware of these tendency patterns can provide important confirming clues that a piece of writing likely came from ChatGPT versus a human.

Quoting Non-Existent Sources

  • When prompted to provide citations for facts and quotes, ChatGPT will often generate fake sources and references. It invents fake researcher names, publication titles, dates, and other bibliographic details that sound real but do not exist.
  • Carefully fact-checking any citations ChatGPT provides against external sources almost always reveals its invented references. For example, scanning a cited academic journal or checking for an actual author by that name exposes the deception.
  • This tendency to fabricate sources stems from ChatGPT’s lack of real world knowledge and lack of memory. Yet the citations can look convincingly real unless properly verified via cross-referencing.

Using Wrong Names and Details

  • When discussing real-world facts and entities, ChatGPT will sometimes use slightly incorrect names or fictional details. For instance, it may state Winston Churchill was the British Prime Minister during WWII, when it was actually Churchill.
  • Fact-checking entities mentioned against trustworthy external sources can uncover subtle inaccuracies and name errors like this. These mistakes point to ChatGPT’s lack of factual knowledge even as it tries to sound authoritative.
  • Other examples include getting historical dates wrong, using a different company name than the real entity, or combining details across unrelated people. Targeted fact verification exposes ChatGPT’s ignorance.

Illogical Arguments and Reasoning Gaps

  • On the surface ChatGPT can produce arguments and analysis that seem coherent and logical. However, upon deeper inspection, there are often major gaps in reasoning, contradictory statements, and logical fallacies.
  • The arguments fail to hold up when scrutinized critically because ChatGPT has no actual comprehensive understanding of what it is discussing. It stitches together plausible-sounding statements without true comprehension.
  • Looking for lapses in reasoning flow, contradictions, and fallacies takes more work but almost always reveals cracks in ChatGPT’s arguments. Human logic is far stronger under careful examination.

Incoherent Responses to Follow-up Questions

  • When asked a series of follow-up questions that build on context, ChatGPT will frequently become confused and contradict itself. Its lack of true contextual memory becomes evident.
  • For instance, if asked for details about a fictional character ChatGPT invented, it cannot consistently answer questions about the character’s background, relations, motivations, etc. The backstory will be muddled and contradictory.
  • Skilled questioning that requires maintaining logical consistency reveals ChatGPT cannot do so for long. It has no actual understanding of the narrative or context to draw upon.

Detection depends on spotting these tendency “seams” in AI-generated text. While ChatGPT fools in isolated exchanges, patterned skill gaps become visible under consistent inspection, cross-verification, and logical questioning. Maintaining skepticism takes work but protects against deception.

Comparing Contexts and Factual Consistency

One powerful technique for detecting ChatGPT’s involvement is to compare separate excerpts of writing and look for contradictions and inconsistencies. This exploits ChatGPT’s fundamental lack of actual knowledge, facts, memory, or consistent context.

For instance, you could prompt ChatGPT to write a short poem reflectively describing its childhood and upbringing. Then in a separate prompt, ask it to summarize where it was born and grew up.

The geographical locations, family details, and other facts will likely differ between the two excerpts as ChatGPT has no genuine consistent identity or history to draw from. It makes up details anew each time.

Similarly, with a supposed fictional story, you could compare descriptions and backstories of the same characters or settings across different chapters or scenes. Inconsistencies in names, relations, backgrounds and timelines will emerge.

Even when asking ChatGPT direct factual questions, it will often contradict itself if asked again separately. The isolated prompt sessions prevent it maintaining consistency.

Therefore, gathering multiple excerpts and thoroughly cross-checking for self-contradictions, inconsistencies, and alignment issues can powerfully indicate an AI attempting to fake contexts, memories and facts it does not actually possess consistently.

You can also verify any real-world facts mentioned against trusted external sources to try exposing inaccuracies. Details like historical dates and figures should match established facts if human-written.

ChatGPT will often inadvertently misalign its fabricated details with reality under cross-verification. its lack of actual knowledge shines through.

With vigilance and comparisons, ChatGPT’s continuity gaps and ignorance reveal themselves through errors a consistent human writer would not make. Checking for contradictions across excerpts provides strong clues about actual authorship.

Wrap Up

Evaluating authorship attribution remains an imperfect art even with the assistance of methods like the ones outlined. However, thoughtfully combining language analysis, testing through re-prompting, assessing prompt adherence, and cross-checking factual consistency can help develop a reasonable assessment of ChatGPT’s involvement.

Keep in mind that human writers also utilize AI tools in their work to varying degrees. The goal is not to definitively prove ChatGPT was solely used, but to critically examine if it significantly contributed to or generated the writing without proper attribution.

By upholding ethics and transparency in how we employ AI in writing and education, we can continue advancing these technologies responsibly and benefiting from their tremendous potential. Discerning ChatGPT’s role through thorough analysis ultimately helps promote its wise and honest application.

Key Takeaways

  • Assess writing style, complexity, coherence, and personality compared to ChatGPT’s tendencies
  • Test excerpts by re-prompting ChatGPT and comparing outputs
  • Watch for over-adherence to prompting instructions and context inconsistencies
  • Note patterned errors like fake citations and incorrect factual details
  • Compare contexts across excerpts – inconsistencies indicate AI generation
  • Skillful combination of these techniques allows reasonable ChatGPT detection
  • Goal is not definitive proof, but estimating its contribution to writing
  • Responsible AI use requires proper attribution and transparency

Further Resources

Writing Style Analyzer

https://analyzemywriting.com

Allows assessing key metrics like readability, word choice, syntax complexity and more to compare human vs. AI writing styles.