Generative AI Aiding Accessibility, Quickly

When I write blog and social media posts on Bluesky and Mastodon, I want a tool to analyze a supplied image and generate concise and accurate descriptions, called "alt-text," to benefit visually impaired readers and help me streamline the publishing effort.  Thankfully, generative AI tools have expanded their ability to process and interpret images, making them ideal for this use case. 

Not All LLMs Are Created Equal

The number of available LLMs has surged, each with distinct features and capabilities. Their performance, however, can vary significantly, even when you use the same prompt and image. These differences are often attributed to the “black box” nature of LLMs, i.e. differences between their internal structures and how they are trained.

Choices, Choices. Which One Should I Use?

I processed a set of test images with prompts across multiple models, then evaluated the generated alt text for accuracy and presentation. This subjective critique helped me choose the best model to meet my needs. You also learn which model is best in class regarding image processing and recognition.

Evaluating LLMs for Alt Text Generation

I selected four LLMs capable of image processing and recognition:

1. OpenAI ChatGPT-4.0
2. Cohere Command R Plus 08/2024
3. Anthropic Claude 3.5
4. Google Gemini 1.5

All models were tested in their free versions without subscriptions.

Prompt Engineering for Alt Text Output

Prompt engineering techniques ensured consistent output over multiple test runs and meaningful comparisons between the models. While I crafted a universal prompt for ChatGPT, Claude, and Gemini, I designed a distinct prompt for the Hugging Chat hosted Cohere model due to its reliance on an external tool, CogVLM, for image processing. 

Example Prompt for Cohere:

You are a skilled social media manager and blog author. Use the CogVLMv1 Image Captioner tool to analyze the uploaded image. Rewrite its output into a concise and descriptive alt text for visually impaired readers. Provide factual image description. Include objects, background, interactions, gestures, poses, visible text, frequency. Describe colors, contrasts, textures, materials, composition, focus points, camera angle, perspective, context, lighting, shadows. Avoid subjective interpretations, speculation. Exclude introductory text, comments and administrative details. Write in English using its active voice, limited to five sentences. If the tool cannot be invoked, state: “Image Captioner tool not invoked, please try again in a new chat.” If the tool returns an error, relay the error message verbatim for troubleshooting.

Example Prompt for ChatGPT, Claude, and Gemini:

You are a skilled social media manager and blog author. Analyze the supplied image to create concise and descriptive alt text for visually impaired readers. Provide factual image description. Include objects, background, interactions, gestures, poses, visible text, frequency. Describe colors, contrasts, textures, materials, composition, focus points, camera angle, perspective, context, lighting, shadows. Avoid subjective interpretations, speculation. Exclude meta-text, comments, or administrative details. Write in English using its active voice, limited to five sentences. If image processing fails, state: “I could not decode an image. Please try again.”

How Did They Compare?

1. ChatGPT 4.0 - ChatGPT excelled in accuracy and descriptive quality, delivering alt text that was clear, concise, and contextually appropriate. It consistently met my expectations and outperformed others in execution speed, making it the top performer.

2. Cohere Command R Plus - Cohere demonstrated strong performance, producing alt text comparable to ChatGPT. However, its reliance on the external CogVLM tool added complexity, and its execution speed was noticeably slower than ChatGPT.

3. Anthropic Claude 3.5 - Claude’s output was solid but fell short of the top two models. Its alt text tended to adopt a third-person tone, such as “The image depicts people doing stuff and enjoying themselves,” which felt less natural compared to ChatGPT’s more direct descriptions like “People doing stuff and enjoying themselves.”

4. Google Gemini 1.5 - Gemini ranked last. While it handled some images well, I noticed it occasionally hallucinated—generating descriptions that didn’t match the image. Additionally, this model refused to process images containing people, a significant limitation for creating alt text for diverse content.

ChatGPT Leads the Pack

From a qualitative standpoint, ChatGPT-4.0 and Cohere emerged as the image processing and recognition frontrunners. However, ChatGPT’s faster processing speed and ease of use gave it the edge in overall performance.

Generative AI makes social media and blog content more inclusive by helping improve accessibility for visually impaired readers while decreasing the publishing effort for creators.

Embedding a Mastodon timeline

One notable feature of the social media platform formerly recognized as Twitter is the capability to integrate your post timeline into a static webpage or blog. In light of the recent user exodus from Twitter to the Fediverse, individuals who had previously embedded their Twitter posts into their blogs found themselves in search of an analogous feature. I was among those seeking a comparable solution.

After some searching on the internet, I stumbled upon a GitLab project 'Mastodon timeline feed widget'. This project is free and open-source and employs JavaScript to make API calls to Mastodon. The script is designed to dynamically generate a post timeline within a web browser and comes bundled with accompanying CSS code to format the resulting widget elegantly.

See the Pen Mastodon embed timeline (open to see it working) by idotj (@ipuntoj) on CodePen.

Free to customize

I operate an Android phone, and my client of choice for the Fediverse is Fedilab. When a user shares an article to Mastodon via this client, Fedilab automatically inserts a markdown '>' symbol into the article's subtitle. Interestingly, Fedilab also converts this markdown symbol into HTML blockquote elements, a functionality not supported by Mastodon's 4.1 web server and user experience (UX). 

FOSS flexibility

Eager to bring this same user experience to my webpage, I forked the original project in early 2023 and developed a markdownToHTML() method. This function parses the post's HTML content, identifies blocks prefixed with the '>' symbol, and encapsulates them with HTML blockquote tags. Subsequently, this enhancement was integrated into the upstream GitLab project.

See it in action

You can see the widget in action here on my personal blog, and at Support Roaring Rock Park's website.

Clean up toxic Twitter and Tumblr feeds

It is no secret that social media has become increasingly toxic. It is now being used less to connect people, share ideas and promote constructive dialog and more as a tool to drive wedges between people and used by actors to spread falsehoods for Machiavellian ends.

So what can you do ?

As this Medium post suggests, you can avoid using social media altogether. But for me, who has many people in my social circles, built up over the years and spread across different time zones, social media engagement allows me to stay in contact with people who matter in my life. So the problem I want to solve is how to filter out content that I find objectionable, thus decreasing the amount of content I find offensive and increasing my enjoyment of reading my social circle's posts.

The following solutions reduce my mental stress of being forced to read posts that contain content that I rather not see.

What social media sites allow you to filter content 

I use several social media sites to stay in content i.e., Facebook, Instagram, Twitter, and Tumblr. The latter two sites allow users to filter content, tuned to words you deem objectionable.

I use these sites' tools to clean up social media toxicity.

How to clean up Twitter  

Mute specific content

In addition to the usual and customary tools to unblock unwanted content (blocking users, muting users), Twitter has an additional tool: a "mute word" feature. The benefit here is that if you identify a common word that you find objectionable, then Twitter's algorithm filters (i.e. it does not show) tweets and replies that contain this word.

You can enter a word in either the Twitter desktop or mobile apps. Here is a quick walkthrough on how to enter a word to be muted by Twitter. I'll step through this process using the desktop as an example. You can take similar steps on Twitter's mobile apps.

1. Go to your Twitter Home Timeline,
2. On the left hand screen, click on "More" -> "Settings and Support" -> "Settings and Privacy". Your Twitter Home Timeline will disappear and you now will see your Twitter accounts settings dashboard
3. Click on "Privacy and Safety" in the middle column
4. In the right-hand column, click on "Mute and block" -> "Muted words". Now the right-hand column will change to reveal where you can enter new muted words and those you may have already specified.
5. Click on the "+" sign in the upper right-hand section of the right-hand column. The right-hand column will change again to allow you to specify a word, phrase, @twitter-account, or #hashtag, that you want filtered out. There are also options you can specify for how long you want this content to be filtered and how Twitter should handle notifications.

Rearrange, and selectively filter, your Twitter desktop's layout

One of the unattractive sections in the desktop's layout is the "What's happening" section. I have found that Twitter bots drive what appears in this section, and I have found some of that bot activity objectionable.

I discovered a desktop browser extension, available for both Google Chrome and Mozilla Firefox, called "Tweak New Twitter". This extension rearranges the Twitter home timeline based on options you select. There are many, many options that you may use. I have found the first eight to be the most valuable.

Interesting note: the "Mute quoting of specific tweets" and "Hide tweets quoting accounts you've blocked and muted" are complementary features of this browser extension, related to Twitter's "Mute Word" feature described above.

One downside: using this browser extension will only improve your Twitter reading experience on a PC or iMac, as it only works with the desktop Chrome and Firefox browsers.

How to clean up Tumblr

Like Twitter, Tumblr also provides its user base tools to selectively filter content based on specified words, phrases or hashtags.

Also, Tumblr has the ability for content creators to tag "Community Labels" on the posts they create. Users who consume that content can either view the content, see the content blurred, or not see it at all. The content labels fall under three broad categories: drug addiction, violence, and sexual themes. Given Tumblr's long and tortured past with adult content, Automattic, Tumblr's parent, has been setting expectations that their introduction of the community label "sexual themes" will not signal a return.

Here are Tumblr help pages outlining how to use its content filtering tools and community labels.

A cleaner reading experience

No solution is perfect. I occasionally see objectionable content, especially if the words I specified to be filtered are embedded in meme graphics. That said, I am more enjoying reading my social media feeds now that these filters are in place.