Worth a try? A look at 14 readability analysis tools
Features, privacy, and AI ethics of 14 readability tools, and why I selected 7 to evaluate for my 'ethical shoestring' writer's toolkit.
As part of my 2024 quest to become a more readable writer, I surveyed 14 readability analysis tools and chose 7 to evaluate. This post shares my survey results. It includes tool features and user experience, use of ads, AI ethics, and privacy.
My main post on readability metrics, tools, and evaluations is here (coming soon!). Four bonus pages provide details:
Readability features of Microsoft Word and why I’m looking for a new tool.
What “readable” means, 16 metrics, the 6 I chose to evaluate, and why.
14 readability analysis tools besides Word, the 7 I chose to evaluate, and why. [YOU ARE HERE]
Scoring for Word and the 7 selected tools on the 6 selected metrics. (coming soon!)
Acronyms:
ARI = Automated Readability Index (readability metric)
ASL = Average Sentence Length (words per sentence)
AWL = Average Word Length (characters per word)
FKGL = Flesch / Kincaid Grade Level (readability metric)
FOG = Gunning Fog Index (readability metric)
FOR = FORCAST (readability metric)
FRE = Flesch Reading Ease score (readability metric)
SMOG = Simple Measure Of Gobbledygook Index
PHW = Percentage of Hard Words (hard words / total words)
UX = User Experience.
Page Contents:
1. Readability Tools - 7 considered and evaluated (FREE).
1.2 Datayze (styled “data·yze”)
1.5 TextCompare “Readabilit” (TCR)
2. Readability Tools - 7 not considered (not evaluated).
Note: Privacy isn’t a big concern to me for these tools. I’m only using readability scoring on texts I intend to publish publicly anyway. Checking AI ethics on these companies is mostly a matter of principles and policy.
0. My Baseline: MS Word
I am using Microsoft Word (retail licensed copy of Office 2019) as a baseline only. Why? 3 main reasons:
It only supports Flesch metrics and requires two UI interactions - Word Count feature and Readability Statistics feature (separate UI buttons and result dialogs).
Its displayed ratios aren’t consistent with its numbers.
It’s inaccurate on native rich text DOCX content, especially if it has URLs. It needs plain text input (in a Word file) to work correctly.
Further details on my experiences with using Word are published here.
1. Readability Analysis Tools - 7 Considered and Evaluated (FREE)
Unless otherwise noted below,
Each tool is a free website with ads (banner, embedded, or popup).
Login is not required.
AI is not known to be used in the readability metrics computations.
For each tool, I summarize Basic features and UX, Use of ads, AI ethics and privacy, and Bonuses. Bonuses are tool features beyond readability metrics which may be of interest to writers.
Privacy note: Even if the metrics don’t use AI today, a tool provider could add AI in the future. And some tools might already use AI “under the hood” for purposes like counting syllables, or for other tool features. It’s good to be aware of what data tools collect and what they do with our data.
If a tool is free, “you’re not the customer, you’re the product.” (Tom Johnson, 2001)
The writing I plan to scan with these tools isn’t confidential or proprietary. If your needs are different (or my needs change), these free tools may not be good choices. See “Ethical Shoestrings” for more on ethics of other types of free tools I use.
1.1 Character Calculator (CC)
Basic features and UX: For Character Calculator, 5 UI interactions (load page, paste text content) are needed per post to get all measures and underlying data for the evaluation.
Character Counter (data inputs).
Flesch Reading Ease and Flesch-Kincaid Grade Level (2 related metrics).
Syllable Counter (data inputs).
Total clicking could be reduced by picking fewer metrics to track. No clicks are needed to trigger the calculations after pasting the text.
As a plus, CC displays both a score (e.g. “11.86”) and a grade level (e.g. “7th grade”) for SMOG. Other than RF, other tools only display the SMOG score, implying the score is the grade level. That may be true at higher grade levels, but not all levels.
Use of ads: On the main page, CC uses ads in the left and right sidebars, between the scores and the explanations, and across the foot of the page. Visual disruption is above average. Thankfully, the other 4 pages do not load ads.
AI and privacy:
CC’s privacy policy says they “currently collect and process only cookies, log files, IP addresses, timestamps, and user agents.”
They use Publift to “provide monetisation technologies for our site”.
Information is “securely stored in the servers in the USA and the website is operated from the Republic of Turkey”, and it is automatically deleted after one week.
Bonuses:
Estimates reading and speaking time in whole minutes.
The CC site supports multiple languages for the UI (although it does not appear to support analyzing any languages other than English).
The CC site includes features such as “Font Changer”, “Word Counter”, “Binary Translator”, “Case Converter”, “Numbers to Words”, “Abjad Calculator”, “Gematria Calculator”, “Words to Characters”. It also includes some for-fun tools: a “Drunk Text Generator” and a “Pig Latin Translator”.
1.2 Datayze (styled “data·yze”)
Basic features and UX: The free Datayze Readability Analyzer includes Gunning Fog Index and SMOG Index, along with the classic Flesch metrics. Datayze is the only tool supporting SMOG that still enforces the traditional minimum of 30 sentences. This suggests that the tool may still use the original sampling method and is not scoring the entire text.
Under the tab “Other Readability Tools”, Datayze offers a Difficult and Extraneous Word Finder and a Passive Voice Detector with a “by zombies” test recommended by Grammarly 😏. Their Difficult Word Finder includes counts of Rare Words, Long Words, Hedge Words, and Adverbs.
“Long” words appear to be “complex” 3+ syllable words. The count is much lower than pure counts of 3+ syllable words in other tools, though. Criteria may be a combination of a word list and syllable count? (But they count “every” as a complex long word, and it’s hardly unusual or complex vocabulary.)
2 more UI clicks are needed to bring up the “Other Readability Tools” tab to get either of these bonus results.
Their privacy page says that “Some of data·yze's apps have a per user daily usage cap.” I did not run into their usage caps while testing for this evaluation.
Note: Datayze offers a file upload option. It accepts a DOCX file - but pulls it in as binary garbage.
Use of ads: Datayze page ads are mostly “below the fold” in my browser and not intrusive. A banner ad often pops up at the bottom. However, whether I’m on the tab or away, it periodically pops up ads (often for Grammarly!) which grey out the screen and require a click to clear. This even happened when viewing the Privacy page!
AI and privacy: The Datayze Privacy policy says it “collects and stores standard anonymized web traffic information, as well as some app input” and may keep it for up to a year. “Data collected by data·yze is used for website improvement. Data collected and stored by data·yze is never sold, or transferred to third parties.” This policy regarding “app input” and using it for “website improvement” could present a potential future risk for AI exploitation. The site gives no corporate location information - however, the founder (Dr. Sarah Tyler) appears to be based in California.
Bonuses: Datayze offers a free Spell Checker, Word Analyzer, Alternate Spelling Finder, and Word Blender.
1.3 Grammarly (free plan)
Basic features and UX: Grammarly is a market leader for spelling, grammar, and style support. It offers a free plan and paid plans starting at $12/month which support readability scoring. The free plan has limited capabilities. Setting up an account and logging in is required, even for the free plan.
Grammarly only shares limited data (characters, words, sentences) and metrics (ASL and AWL) in its reports.
Grammarly’s basic Performance scoring does not show the percentage of passive voice sentences. This is somewhat odd since Datayze references a Grammarly page on passive voice and the “zombie test”.
Flesch is the only type of readability metric Grammarly supports.
FRE score is in the Performance section (under the Overall Score).
Grammarly does not provide FKGL as a number, but gives an approximate grade level in text in the Performance dialog.
Use of ads: Its only ‘ads’ are for Grammarly (upgrades to premium). It promotes upgrades to AI features by offering a few uses free:
Clicking ‘Dismiss’ does not suppress further prompts. The upgrade prompts cannot be turned off on a free plan. But they can be ignored.
AI and privacy: There’s no sign that AI is used for readability scoring. However, Grammarly openly promotes that their tool uses AI.
I can easily avoid using Grammarly’s AI-driven features. However, it seems impossible to prevent them from using scored texts to train their AI models (even if I had a paid plan).
Their Privacy and Security FAQ says they use AWS multi-tenancy to ensure that writing suggestions from one customer’s content are never offered to a different customer. It also says they “may retain random samples of de-identified text to help us improve the algorithms underlying our software and services” but “do not share the texts that you check with Grammarly with any third parties” and “have not sold, do not sell, and will not sell customer data”.
It’s hard to believe that an individual free user would get their very own AWS tenant. The safe assumption is that all free users are likely pooled in the same AWS tenant. Texts from free users are probably used to generate writing suggestions for any customer (free or paid).
I’ll continue monitoring their Responsible AI policies and standards.
Bonuses:
Estimates reading and speaking time to the second (shown in the screen shot above).
Supports a personal dictionary.
1.4 Readability Formulas (RF)
Basic features and UX: Readability Formulas offers a very complete set of readability metrics with a single UI interaction. It is the only evaluated tool that supports FOR.
RF provides exhaustive details on the counts and derived measures. The results include counts for words of 1, 2, 3, 4, 5, 6, and 7+ syllables. In contrast, most tools only show one count of “polysyllabic” words (3+ syllables).
Full results can be printed to PDF. (It also has a ‘copy to clipboard’ function, but it creates TXT content with way too many line breaks.)
RF’s transparency is excellent. For each metric, it shows the formula and the values that went into it. (It does not always show how a score is mapped to a grade level, however. Most tools don’t.)
Like Word, RF automatically calculates the percentage of sentences in passive voice. (Datayze can also calculate it with a few additional clicks.)
RF is configurable and provides help on the impact of its configuration settings. Oddly, the default for compound sentences (treat as one) does not match what the help text recommends (treat as two). I did not change any defaults for this evaluation.
By default, RF counts both “sentences” and “lines”. (Line counts include sentences that don’t have ending punctuation). Results indicate that if RF detects 20% more lines than sentences, it defaults to using the line count in the readability metrics (this is configurable). This seems advantageous for texts (like mine!) which include section titles, bulleted lists, numbered lists, and references. However, other configuration options are available.
One potential downside is that RF has a text limit of 3500 words.
I would prefer if it were unlimited or had a higher limit, but this is not a blocker. I already decided a few months ago to look hard at splitting draft posts over 2500-3000 words. I haven’t published a post over 3000 words since then, although this one is tracking close to 4000 :) I can always run the tool twice on a very long post, if needed.
Use of ads: RF has a large banner ad across the top of the page, and a smaller ad in the left sidebar below the formula selector. Once you scroll to the part of the page you want to use, they’re not disruptive. Like Datayze, RF has periodic gray-screen popups that mostly promote Grammarly.
AI and privacy: RF’s privacy policy isn’t linked to the home page and was only findable by searching the site. However, the contents seemed reasonable. They “do not use cookies to store any information you submit”, “do not collect personal information”, “do not connect to any databases to transmit files or information”, “do not store any information on a server”, and “do not sell or rent your personal data to third parties”.
Bonuses:
RF offers a companion Spanish Readability site that assesses readability of texts written in Spanish. They offer over 10 readability metrics to choose from which are designed for Spanish!
RF offers Writing Tips and other resources on readability. Examples include:
Advice on supporting ‘impaired readers’.
Tips on creating a ‘reader profile’.
Why writing for 8th grade level is generally a good idea.
Detailed word counts include: easy, hard, unique, repeat, compound, abbreviated, and misc words; proper nouns; and cardinals in text.
Additional tools can be run on the same text that was scored for readability. These tools can run on the same text with a few more UI clicks.
Fry Sight 1000 Words Assessment rates the minimum grade level for a reader. Grade level is based on use of specific words in a 1000-word list.
Example results from running the Fry Sight 1000 Words Assessment Lexical Density and Diversity analysis of writing style (info here).
Example results from the bonus Lexical Text Density Analysis and Lexical Word Diversity Analysis tools
One minor concern: I’ve noticed multiple small errors in the formula images on the documentation pages for the metrics. (Example: misplaced parentheses on the SQRT term in the second image for SMOG readability.) I’m planning to notify the company. In the meantime, don’t rely on their pages as authoritative about the formulas without verifying them.
1.5 TextCompare “Readabilit” (TCR)
Basic features and UX: TextCompare is the home site for a small collection of text analysis and processing tools. The Readabilit sub-site does not support FOR, but requires only 1 UI interaction to get all other metrics selected for this evaluation. It also supports 6 of the 10 readability metrics I did not select (more than any other free tool). The page also provides helpful tables for interpreting grade levels for all metrics they support. One downside is that results are on two tabs, requiring more clicks.
Use of ads: The TextCompare site uses large banner ads at top, middle, and bottom of the home page. However, the Readabilit page is blessedly free of ads.
AI and privacy:
The TextCompare home page says that no content entered is sent to their server.
The Readabilit Help page and Privacy page say that information uploaded to their server is not shared with any third party.
These two points are somewhat contradictory. The corporate relationship between TextCompare and Readabilit is not clear. Neither site’s “About” page provides any useful insights.
Bonuses:
TCR is the only free tool that estimates writing time, as well as reading and speaking time, to the second.
TCR includes a results tab which checks for various types of word usages, such as buzzwords or what it calls ‘profanity’. (It flags as profane words one might not expect, though - such as “cancer, dive, fight, illegal, poverty”.)
1.6 WebFX Read-Able
Basic features and UX: WebFX is the home site for a SEO-oriented toolset. Read-Able can read from a URL or pasted text. HTML is allowed and will be stripped.
WebFX supports ARI, Gunning, and SMOG as well as the Flesch metrics. One UI interaction scores all of them.
If you want to see the details on how WebFX calculated the values, you have to click “See details” under each individual formula. (Since the details only add one additional line of text per formula, it would be much nicer if details were always displayed, or if one click could ‘expand all’.) It displays counts for complex and polysyllabic words.
Like Word, WebFX doesn't display the syllable count it uses in the Flesch formulas. However, unlike Word, WebFX provides enough other data that it can be back-calculated.
Use of ads: No third party ads were seen on the site while using it for this evaluation. The page includes links and information about SEO offerings and other WebFX features.
AI and privacy: The WebFX policy page says they “may disclose Personal Information with companies that are affiliated with us”. The footer of the WebFX site says:
It is not clear how TeamAI (“Gemini, GPT-4 and LLaMA | TeamAI's Shared AI Platform”) is connected to WebFX. They may share users’ data with TeamAI.
Their site does not honor “do not track” signals from browsers.
WebFX is based in Harrisburg, PA, USA. The page refers to rights based on “state” and has no express indications of support for GDPR regulations.
Bonuses: In addition to input from URL or pasted text, WebFX supports a “web reference” which can embed readability scoring into a link on a web page. This might not work well on a Substack post page, though.
Their site includes an “Emoji cheat sheet” and a “Color picker”.
1.7 WordCalc
Basic features and UX: The main WordCalc page is a Syllable Counter. The WordCalc Readability Calculator only requires 1 UI interaction to get all selected metrics (except FORCAST and Linsear, which it doesn’t support). The UI is easy and fast. The Syllable Counter requires a separate UI interaction, including pasting the text again, and only provides syllables, sentences, words, and “letters”. It’s a little slower, but still completes within a few seconds.
One major downside is that the readability scoring is not transparent. No formulas or shown and no ratios are provided with the Readability Calculator to build confidence in the metrics or to give insights on what to change to improve readability.
Use of ads: WordCalc has banner ads at top and bottom, and uses the same full screen popup ads as other sites in this report.
AI and privacy: WordCalc’s privacy policy says the site is owned by Ezoic (HQ in Carlsbad, California). It points visitors to the Ezoic privacy policy. They collect fairly intrusive information about the device you’re using with their tool, and save geolocation data. They say they do not share information with others except as the policy describes. However, they have a blanket statement about sharing with “partners” who provide services. This seems to leave the door open to sharing data with an AI platform provider in future. They also say they share with “companies within our corporate family” and they assert that “We may store your information in an aggregated and anonymized format; we may use this information indefinitely without further notice to you.”
Their policy also states that “We comply with the EU-U.S. Data Privacy Framework ("EU-U.S. DPF"), the UK Extension to the EU-U.S. DPF, and the Swiss-U.S. Data Privacy Framework ("Swiss-U.S. DPF") as set forth by the U.S. Department of Commerce.”
To their credit, they have a explicit section on “Third Party AI Services”. It names Stability.ai Stable Diffusion, OpenAI, ElevenLabs, Amazon Polly and Google Gemini as potential services. They say “Where applicable, we opt out of sharing data with, or allowing such third-parties to use our data.”
The safe assumption with WordCalc is that your data will be kept and shared and made available to someone’s AI system at some point, if not now.
Bonuses: None noticed.
2. Readability Analysis Tools - 7 Not Considered (Not Evaluated)
I dropped some interesting tools from my evaluation because they aren’t free. Others were set aside due to limitations in supported metrics or text capacity. For all tools not considered, I didn’t bother to dig into if or how they are using AI, or whether they are sourcing and using data ethically.
Note: Readability scoring features are also available in sophisticated document authoring and workflow tools, such as ClickHelp. However, ClickHelp is overkill for my writing. And at a starting price of $175/month, it isn’t an option on a “shoestring” budget. No tools in this class are listed below.
2.1 Python tool
Alternative to consider if no free tool works out: Writing my own readability checker in Python using py-readability-metrics. Omitting for now; although I still love writing code, I’m not a fan of reinventing the wheel.
2.2 GoWinston.ai $
3 concerns quickly surfaced with GoWinston.ai’s readability score checker.
Not free (only 7-day free trial).
Doesn’t support Flesch-Kincaid Grade Level.
The tool threw up a warning on Case 1 about my text possibly being AI-written (definitely not). Looks like a sales pitch for their other paid features?
2.3 Hemingway App $
Hemingway App is primarily a grammar and style tool. It uses ARI (only) to assess a grade level for writing. The base Editor is a flat $19.99 for macOS or Windows, runs offline, and has no AI. They offer a 2-week free trial.
Editor Plus is an optional paid add-on subscription. It allows configuring a target grade level and uses AI to help people revise their text. (I did not review their AI policies since I chose not to pursue using Hemingway.)
2.4 Originality.ai $
Originality.ai supports all 6 of the selected metrics, and 5 of the 10 unselected metrics. It requires 3 UI interactions to get all of the values - not the best, and not the worst.
Their readability checker page doesn’t require login and doesn’t have an obvious limit to how many times it can be used. However, I inquired to confirm if their readability tool is free for long-term use. It’s not.
Response from “Ryan”: “Hi there, Thanks for reaching out. So our readability does require you to have a credit balance in which you can purchase at $0.01. Each credit will allow you to scan up to 100 words per credit. Also, for the readability scanner, we do have a 3,000 words per scan limit. So if your content is a bit longer than that then we recommend for you to scan in parts in order to avoid reaching the word limit.”
2.5 Pro Writing Aid $
Pro Writing Aid offers a free plan which requires an account to use. After creating a login, I decided to skip the tool.
It calculates Flesch-Kincaid Grade Level but only calculates Flesch Reading Ease per paragraph, not at the document level. Paragraph level is useful and a plus, but I want the overall document score.
The only 2 other readability metrics it supports are ARI (selected) and Coleman-Liau (not selected).
A free account only supports scoring texts up to 500 words. That’s too short to be practical for my writing.
2.6 Readable $
Readable offers a comprehensive set of metrics (15, all but Linsear Write). It also has good documentation on readability formulas it supports. However, it is not free; it only has a 7 day free trial. Paid plans range from $4-$69/mo.
2.7 SiteChecker Pro
SiteChecker Pro is free and provides all supported measures with 1 UI interaction. However, it only supports FRE and Gunning Fog, and does not provide FKGL for comparison or use.
3. Summary
The Datawrapper table below maps the 14 tools I reviewed to the 6 selected readability metrics they support. The table also includes the number of browser UI interactions needed to get all of the selected metrics and supporting data. Each browser UI interaction includes opening a specific web page and pasting TXT file content. Some tools need an extra click to start the analysis. When tools provide results on multiple tabs, more clicks are required.
In the evaluation, I will track, but not rely on, bonus metrics such as percentage of passive voice sentences, reading time, and speaking time. None of the tools which calculate them give formulas or explain how the values are calculated. They are interesting, but not actionable.
Next step? Use these selected tools on some of my past writing to see how well they work and what the readability metrics look like. See Scoring for Word and the 7 selected tools on the 6 selected metrics for the write-up on those findings!
4. References
Related Posts
TO DO: Add embeds for the last 2 related posts once they are published
Main post on readability metrics, tools, and evaluations - here
Scoring for Word and the 7 selected tools on the 6 selected metrics.
What “Readable” means, 16 metrics, the 6 I chose to evaluate, and why.
Readability features of Microsoft Word and why I’m looking for a new tool.
![Using MS Word for Readability Analysis [Bonus page]](https://substackcdn.com/image/fetch/w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F592d16c2-c2e9-4e79-a821-8e1f24d2a895_1165x599.png)