June 16

AI Book Summarizing Tools are Really, Really Bad (Right Now)

ChatGPT-like tools are pretty good at summarizing an article or webpage, but don’t ask the technology to tackle a whole book (yet). My tests reveal that these tools don’t actually read the books. They know basic ideas about the book and make sh— uh— stuff up, elevating them into the category of liar-liar-pants-on-fire.

Trying Out AI Book Summarizing Tools

For this AI Experiment, I used three books to test the three AI tools listed in the “Book Summaries” category on my new AI tool directory obsession: There’s an AI for That.

The three tools are BooksAI, Booknotes and Summarist. And they all lied and failed.

Test #1: Lies about a hot new book

Cover of the book "Employalty" by Joe Mull. The cover features a trash can with a "Help Wanted" sign discarded in it, set against an orange background. The subtitle reads "How to Ignite Commitment and Keep Top Talent in the New Age of Work." Clearly, AI book summarizing tools wouldn't do this justice!

Speaker Joe Mull just released Employalty: How to Ignite Commitment and Keep Top Talent in the New Age of Work. It is HOT! The book shot to the top of several Amazon categories, and it’s now in airports and bookstores around the country.

I started with Summarist, and Joe told me the summary was generic, inaccurate and pretty much total bunk. Summarist changed the author and even added a section that Joe never wrote.

        Screenshot of a book summary titled "Employalty: How to Ignite Commitment and Keep Top Talent in the New Age of Work." Highlights emphasize employee engagement with annotations like "YES! WOW!" and "UMMM, NO?". This Right Now tool feels tailored for AI Book Summarizing Tools.
A text titled "Talent Development" discusses employee growth and feedback. Another titled "Flexible Work Arrangements" highlights the benefits of telecommuting and flextime. The phrase "GENERIC DRIVEL" is overlaid in red. The documents were reviewed using AI summarizing tools to distill key points effectively.
Text image with a header "Building a Strong Employer Brand: NOT A TOPIC" and an "X" across the body text. The text discusses the importance of a strong employer brand, highlighting that ignoring it could be really bad, especially right now.

Test #2: Shocking revelations about my most recent book

The cover of a book titled "NERD 365: A Year of Opportunities to Upgrade Your Life" by Beth Ziesenis, featuring a cartoon robot holding a pencil. Right now, you can discover practical tips and insights in this engaging guide.

My latest book isn’t HOT like Joe’s, but it’s holding its own. I used Booknotes and was shocked by three things I found. First, it admitted that it doesn’t read the books! Second, it totally made up quotes that it said would match my style. And finally I was utterly surprised that to get it to make a summary, I had to put in my own OpenAI key to use it. The use costs almost nothing, but still. Very odd. And annoying.

A text with the title "Nerd365: A Year of APP-ortunities to Upgrade Your Life" by Beth Ziesenis, with underlined text indicating an AI language model saying it can't read books but suggests notes and key ideas using AI Book Summarizing Tools.
Screenshot of a text box stating a disclaimer about not providing book quotes, followed by a list of five supposed quotes from Beth Ziesenis, with the handwritten text "FAKE QUOTES... THIS IS CRAZY!" on the side. Right now, it's evident that bad AI book summarizing tools might be at play.
Screenshot of a settings page showing fields for entering a license key and an OpenAI API key, with a comment suggesting not to use one's own OpenAI key due to associated charges. Right now, it's critical because the fees can be quite high for AI Book Summarizing Tools.

Test #3: A pretty good summary for a best seller (and yet more lies)

Cover of the book "The Code: The Unwritten Rules of Fighting and Retaliation in the NHL" showing hockey gloves and sticks on ice. Written by Ross Bernstein with forewords by Marty McSorley and Tony Twist, it's a must-read right now for fans wanting more than bad AI book summarizing tools provide.

Hall of Fame Speaker Ross Bernstein has written more than 50 books, but this one is his most famous. It’s the basis for a movie and has sold super well for many years.

I thought maybe Joe’s book summary wasn’t accurate because it’s so new. And my book wouldn’t work well because it isn’t a best seller. But Ross is kinda famous, as is this book. Autographed copies are going for $85!

When I sent Ross the summary from the first two tools, he said, “Yeah, that’s pretty much right.” But then I used BooksAI, which messed up the name of the author and date of publication.

Screenshot of a webpage with text about the book "The Code: The Unwritten Rules of Fighting and Retaliation in the NHL." Highlighted text is marked as incorrect, indicating publication date issues. Right now, it appears the information isn't accurate.

Stay Tuned for Better Results Soon

I’m thoroughly unimpressed with these AI book summary tools. But that’s today. Technology’s lightning pace of advancement these days means that we will see significant improvement within the year (or the month, or the week).


Tags

artificial intelligence, efficiency, utilities


You may also like

Battle of the Songbots, 2026 Edition

Battle of the Songbots, 2026 Edition
{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}
>