AI Writing Co-pilot

Source: DALL-E 3

Generative AI applications like Grammarly are revolutionizing writing, enhancing quality, and boosting speed. Yet, the elephant in the room is data privacy. To sidestep concerns, partnering with services that offer robust data protection is key. Better yet, consider solutions that don't require sending your precious words to a distant server.

Some things are best left unsaid ..

.. and not processed by remote systems outside of your control.

Enter the realm of open large language models (LLMs). These models, unlike their proprietary cousins from OpenAI (ChatGPT) and Google (Gemini, formerly Bard), process data locally, putting your privacy concerns to rest. And, equally important, open source LLM have licensing terms that stipulate the output generated by them is owned by you, and not the creators of the LLM models (Note: always read the licensing terms).

Driven by a need for data privacy, I crafted a simple AI tool leveraging an open-source LLM to spruce up my writing. This journey wasn't just about creating; it was about diving deep into the world of generative AI and LLM customization.

Hello Hal, improve my writing ...

Source: wallpaperbetter.com

As I wrote about in a previous blog post, I have been upskilling in generative artificial intelligence and machine learning. This opportunity allowed me to learn about generative AI application development with a use case that would directly benefit my daily personal and professional life. As I got on with this endeavor, one common challenge quickly surfaced.

.. but, Hal, help in the way I want.

A common issue with using LLM is that if you do not set explicit expectations on how they should generate content, they tend to hallucinate i.e. produce gibberish and unpredictable content. The solution here is employ a common technique called prompt engineering to provide that missing context to the LLM to avoid it going off the rails.

Choices, choices ...

The number of available open source LLM is growing, and they differentiate themselves based on size, performance benchmark scores, and how they are fine tuned to meet specific use cases. Naturally, evaluating these models can be daunting. Ollama is changing the game by:

1. Offering a rich selection of models, all while upholding strict privacy standards,
2. Bridging the gap between open-source LLMs and applications,
3. Simplifying integration with a uniform API.

I find Ollama to be invaluable when integrating AI into applications.

Meta vs. Google

Evaluating open source LLM options, I considered Meta's Llama2 and Google's Gemma.

From a licensing perspective, I found Google's terms more favorable versus Meta's terms. In particular, when I read Meta's license terms, I found it to be silent on the ownership issue of generated output.

From a technical perspective, I used Ollama to pull both models and, using prompt engineering, applied the same system prompt. Gemma produced more predictable and neatly formatted output when compared to Llama2.

Google's Gemma was the clear winner.

The final product

Remarkably, I implemented this application using less than 60 lines of Python code, and a model file that contains the specialized system prompt to the LLM.

If you are interested in the Python source and the system prompt, you can download it from my Github.

Feel free to reach out to me on LinkedIn for questions about this project and how to integrate open source LLM in your projects.