ChatGPT-like tools are pretty good at summarizing an article or webpage, but don’t ask the technology to tackle a whole book (yet). My tests reveal that these tools don’t actually read the books. They know basic ideas about the book and make sh— uh— stuff up, elevating them into the category of liar-liar-pants-on-fire.
Quick Links
Trying Out AI Book Summarizing Tools
For this AI Experiment, I used three books to test the three AI tools listed in the “Book Summaries” category on my new AI tool directory obsession: There’s an AI for That.
The three tools are BooksAI, Booknotes and Summarist. And they all lied and failed.
Test #1: Lies about a hot new book
Speaker Joe Mull just released Employalty: How to Ignite Commitment and Keep Top Talent in the New Age of Work. It is HOT! The book shot to the top of several Amazon categories, and it’s now in airports and bookstores around the country.
I started with Summarist, and Joe told me the summary was generic, inaccurate and pretty much total bunk. Summarist changed the author and even added a section that Joe never wrote.
Test #2: Shocking revelations about my most recent book
My latest book isn’t HOT like Joe’s, but it’s holding its own. I used Booknotes and was shocked by three things I found. First, it admitted that it doesn’t read the books! Second, it totally made up quotes that it said would match my style. And finally I was utterly surprised that to get it to make a summary, I had to put in my own OpenAI key to use it. The use costs almost nothing, but still. Very odd. And annoying.
Test #3: A pretty good summary for a best seller (and yet more lies)
Hall of Fame Speaker Ross Bernstein has written more than 50 books, but this one is his most famous. It’s the basis for a movie and has sold super well for many years.
I thought maybe Joe’s book summary wasn’t accurate because it’s so new. And my book wouldn’t work well because it isn’t a best seller. But Ross is kinda famous, as is this book. Autographed copies are going for $85!
When I sent Ross the summary from the first two tools, he said, “Yeah, that’s pretty much right.” But then I used BooksAI, which messed up the name of the author and date of publication.
Stay Tuned for Better Results Soon
I’m thoroughly unimpressed with these AI book summary tools. But that’s today. Technology’s lightning pace of advancement these days means that we will see significant improvement within the year (or the month, or the week).
