Artificial Intelligence on Junian Dev

Robots.txt to Allow LLM AI Like ChatGPT for Search Only Using Content Signals

author@junian.dev (Junian Triajianto) — Fri, 05 Dec 2025 19:00:00 +0700

Previously, I wrote about how to block all LLM AIs like ChatGPT and then decided to allow them to visit again.

I thought most AI companies didn’t crawl very frequently, but then I checked my Cloudflare dashboard.

Look at the image below. It’s from Cloudflare’s AI Crawl Control. As you can see, the top crawler is ChatGPT. And interestingly, they crawl the most while actual user visits from ChatGPT remain the lowest.

Welcoming LLM AI to My Site

author@junian.dev (Junian Triajianto) — Sun, 27 Jul 2025 16:57:47 +0700

As you may know, I’ve written a tutorial on how to block LLM AI like ChatGPT from crawling my site without attribution via robots.txt file.

I implemented that on my own websites, and so far I haven’t seen any traffic coming from the LLM bots.

Enough time has passed since then, and I’ve noticed that some visitors to my YouTube channel are coming from LLMs like ChatGPT.

They’re a small number for now, but I guess the future of web traffic will come from LLMs.

Robots.txt Rules to Block LLM AI like ChatGPT from Crawling Your Web Content

author@junian.dev (Junian Triajianto) — Tue, 08 Oct 2024 04:29:59 +0000

I recently checked my Google Search Console and found out that some of the visitors are AI agents.

I believe this isn’t human visiting or reading my article. It’s most likely just an AI agent, scraping my article, and providing the summary to their user.

I love the development of AI, especially Generative AI. But, the business practice isn’t fair yet. How can humans be motivated to write something original if it doesn’t get any benefit at all?

Installing AUTOMATIC1111 Stable Diffusion Web UI on macOS with Apple Silicon

author@junian.dev (Junian Triajianto) — Wed, 02 Oct 2024 15:46:44 +0700

I want to play and research about Generative AI, especially image generation with Stable Diffusion, but I don’t have an NVIDIA-powered GPU.

Luckily, my MacBook Air, powered with M1 Apple Silicon, is powerful enough for Stable Diffusion.

I’m gonna share with you how I’m able to install and run Stable Diffusion Web UI on macOS with Apple Silicon.

Install Dependencies

The first thing to do is you need to have the Homebrew package manager installed. If you don’t know how you can read the guide here.