Tweet
Key Takeaways
- Gemini Bard outperforms PaLM Bard in answering mathematical word problems, with Gemini Bard correctly calculating an answer and PaLM Bard being unable to understand the question.
- PaLM Bard is decent for code generation, but Gemini Bard provides a more detailed explanation of the code, making it a better option overall.
- Gemini Bard excels at text summarization by breaking down the text into digestible paragraphs with bullet points, while PaLM Bard summarizes it in a single paragraph.
Google recently launched Bard with a new language model called Gemini, and with it came a number of massive promises. The biggest was that Google said it would now outperform GPT-4 in most scenarios, and in our comparison of it to GPT-3.5, it does seem to be improved. However, I’m in a unique position where, as a European, I still have to use the old Bard powered by the Pathways Language Model (PaLM). As such, I put the new Bard up against the old Bard to see what improvements were in store, and the results were wild.
Throughout this article, we’ll refer to the old Bard as PaLM Bard and the newer Bard as Gemini Bard. PaLM Bard also says “PaLM 2†in the logo beside its responses.
Mathematics and mathematical word problems
Why are you using an LLM for math?
The first test compared how Gemini Bard and PaLM Bard answered mathematical word problems. As you can see from the above screenshots, I asked both how many burritos tall a 5-foot-11-inch person would be if the average burrito is approximately 5 to 6 inches long. Gemini Bard had no problem with the question and calculated the average person’s height correctly. However, PaLM Bard misunderstands the question entirely, dividing 5.11 (not 5 foot 11 inches) by 6.
While you should never use an LLM for math, at least Gemini Bard understands the question and answers it correctly. PaLM Bard really floundered here, struggling to comprehend the question at all and giving an entirely nonsensical answer.
Code generation
PaLM Bard was quite good for code generation already
I like generating code with Bard, and it’s a big help when trying to reason out a problem I’m having in Python. In this case, PaLM Bard wasn’t too bad, but the example it gave was super basic. It did give instructions on how to run it, but Gemini Bard explained line by line what the code did and provided comments. Both have their advantages, but Gemini Bard did a better job here overall, even if PaLM Bard wasn’t too far behind.
Text summarization
Gemini Bard just does it better
LLMs are good for something as simple as text summarization, but they do it at varying degrees of reliability. Gemini Bard does a great job here by breaking up the text I fed it into multiple paragraphs with bullet points, making it easy to digest. PaLM Bard still gets the details right, but it summarized it in a paragraph, so it’s nowhere near as digestible. It’s not bad, but if I were to pick my favorite one, it would definitely be Gemini Bard.
Of course, an LLM can’t replace reading the whole article yourself, but if you’re pressed for time, then an LLM like Gemini Bard is more reliable than PaLM Bard.
Recipe generation
Let the computer plan out your meals
If you want to plan out a dinner and want to leave it to an LLM to tell you what you can make with the contents of your kitchen, then Gemini Bard will do a better job than PaLM Bard. In my test, Gemini Bard laid out the information in a proper cooking format, whereas PaLM Bard lumped the text together. They both gave viable answers that make sense, but it’s the presentation of PaLM Bard that lets it down.
This isn’t just a one-off, either. PaLM Bard consistently lumps chunks of text together like this. On the other hand, Gemini Bard appears to understand that presentation matters as much as the content of the answer. We noticed the same thing when it came to text summarization, and even in the mathematical word problem, Gemini Bard broke it down into bullet points.
Gemini Bard is a major improvement over PaLM
Unsurprisingly, Bard with Gemini Pro is a major improvement over old Bard. There are really no regressions in performance whatsoever. The worst you’ll find is that it still struggles with math problems, but even LLMs like ChatGPT will hallucinate answers in most cases where a complex question is asked. You’re better off using a tool like Wolfram Alpha instead, regardless. As a result, I don’t take that as a slight against Bard too much.
No matter what, though, Gemini Bard manages to do a lot better than PaLM Bard, even in just how it structures its answers. With a better layout, bullet points, and overall improved readability, it’s hard to say Gemini Bard isn’t a massive improvement. Gemini Pro isn’t even the best version of Gemini (that goes to Gemini Ultra), and you can run some powerful models with any of the best PCs and LM Studio.
** (Disclaimer: This video content is intended for educational and informational purposes only) **
More...
