mizore / notes

gdirect: Direct Google Drive Access from the Terminal — How It Works and Setup Guide

Sun, 25 Oct 2026 00:00:00 +0000

Pulling tens of gigabytes of datasets or backups from Google Drive over a proxy usually burns through expensive data quotas and bottlenecks on proxy node bandwidth; downloading in a browser often drops partway through and forces you to start over. To solve this, I wrote gdirect: a terminal tool for browsing, downloading, uploading, and mounting Google Drive.

This post is organized into three parts: a quick TL;DR on how it "disguises" connections to reach Google directly, a ready-to-use installation and 10-minute illustrated setup guide, and finally a technical deep dive into the underlying source code.

Correction, 2026-10-26: The first version of this post said 183.56.143.147 was a Google google-cn edge node on China Telecom in Guangzhou, so Drive traffic "never physically crossed the firewall". A reader, Ken, wrote in to point out that this was wrong, and we confirmed it: it is a third-party relay (reverse proxy) that forwards traffic to Hong Kong and from there to Google, and the genuine Google nodes inside China do not serve Drive at all. Since 0.7.0, gdirect sweeps the address space Google publishes as its own and finds genuine Google front ends that happen to be reachable. Since 0.8.0 it tries that relay first by default (it is fast and stable), falls back to the sweep results when it fails, and offers --google-only to leave relays out entirely. The post below has been corrected throughout; the investigation is in Correction and Improvements.

TL;DR: How Do We Connect Directly Without a Proxy?

On networks in mainland China, Google Drive's domains (drive.google.com, www.googleapis.com, drive.usercontent.google.com) are blocked. DNS for drive.google.com is poisoned with addresses that belong to someone else; the other two resolve to real Google IPs, but those IPs are blackholed at the TCP level.

Google has far more addresses than DNS ever hands out, though. Google publishes every range it owns (goog.json); subtract the part it leases to Google Cloud customers (cloud.json) and what remains is the address space of Google's own services. Servers there sit behind Google's front-end serving layer (GFE, the Google Front End: the component Google's security whitepaper describes as handling TLS termination, DDoS protection and load balancing), which routes each request to a backend service by its HTTP Host header and holds valid certificates for *.googleapis.com and *.googleusercontent.com. In our tests, some of these front ends are directly reachable from inside China, probably because domestic DNS never hands them out and so they have not made it onto the blocklist.

gdirect takes advantage of this by "opening TCP to a reachable Google front-end IP while sending authentic Google Drive requests (hostnames and certificate checks) over TLS and HTTP":

Point the underlying TCP socket at a scanned endpoint: bypassing poisoned system DNS, gdirect connects straight to port 443 on the target IP. Candidates come only from Google's published address space: TCP first, then full verification only for addresses that answer (official Google certificate + Drive v3 API response + test file download), stopping at 8; results are cached in local SQLite. Since 0.8.0, gdirect tries the third-party relay 183.56.143.147 first by default (fast and stable in practice) and falls back to the scanned Google front ends when it fails; if you don't want any third party in the path, --google-only (or GDIRECT_GOOGLE_ONLY=1) restricts it to endpoints inside Google's space. During socket creation (Rust's reqwest::resolve and a getaddrinfo hook in a patched C++ libaria2), every connection stays pinned to that endpoint, even across multiple cross-domain 302 redirects mid-download.
Keep real Drive hostnames and certificate checks at the upper layers; retry under a "forged identity" when reset: the real hostname is sent by default. If a censoring middlebox resets connections carrying the real SNI (showing up as a connect error or timeout), gdirect automatically retries once under a forged identity: TLS connects as g.cn (a Google-owned name the front end serves a matching certificate for) and verifies the certificate against g.cn, while the HTTP Host header still carries the real Drive hostname such as www.googleapis.com. GFE routes by Host, so Drive requests work as usual, and because the TLS name and the certificate name always match, certificate verification is never weakened. That is the essential difference from "just rewriting the SNI", where the certificate no longer matches. Whichever identity works is remembered in SQLite.
Certificates are anchored to Google's own GTS roots only: no general-purpose CA bundle. Google issues every endpoint we use under its own Google Trust Services roots, so verification trusts only those embedded roots. Even if a third-party CA is compromised or an intermediate key leaks (like the eMudhra key exposure made public in October 2026), it cannot be used to impersonate Google to gdirect. This holds for the relay too: TLS passes through it untouched, so it cannot read tokens or file contents.
Fail over across endpoints: when the current endpoint stops answering, gdirect first tries the other TLS identity, then each other endpoint verified within the last day; dead endpoints are marked in the database, and downloads resume the partial file on the next one.
Separate the control plane from the data plane: sign-in, account verification and token refreshes (accounts.google.com, oauth2.googleapis.com, www.googleapis.com), only a few kilobytes, go through the normal system network (your proxy; since 0.8.0 the HTTP/HTTPS system proxy on macOS / Windows, or HTTPS_PROXY on Linux) so authentication always works; bandwidth-heavy directory listing, multi-connection downloads, and 32 MiB resumable uploads all go through the direct endpoint without spending proxy traffic.

Traffic Type	Commands / Domains	Route
Drive API, downloads, uploads	`browse`, `ls`, `download`, `upload` (`www.googleapis.com`, `drive.usercontent.google.com`, `*.googleusercontent.com`)	The relay `183.56.143.147` first, then scanned Google front ends (`--google-only`: front ends only). Real SNI by default, falling back to the forged `g.cn` identity on resets (`Host` stays real); fails over to the next verified endpoint. `--ip`, `--sni` / `--no-sni` override
Sign-in, account verification & token refresh	`login` and automatic token refresh (`accounts.google.com`, `oauth2.googleapis.com`, `www.googleapis.com`)	System network (system DNS and proxy; honors the HTTP/HTTPS system proxy or `HTTPS_PROXY`)
Disk mount	`mount` (invokes `rclone` under the hood)	System network

Important Note for Users in Mainland China: Sign-in requests need the proxy; transfer requests need a direct path. Since 0.8.0 the easiest setup is to keep your proxy client in rule mode with the system proxy on and TUN off: on macOS / Windows, gdirect uses the system proxy for sign-in automatically (on Linux, set HTTPS_PROXY), while scans and transfers bypass the proxy and connect directly—no direct rules needed. With TUN mode on, the traffic will likely be sent back through the proxy: gdirect connects to Google's own overseas IPs (such as 142.250.x.x or 192.178.x.x), which rules routing by GEOIP or Google IP ranges send to the proxy; the TLS SNI carries a Drive hostname, so a sniffing proxy matches it against the google ruleset and hijacks the connection; and TUN "answers" TCP for every address, so the scan measures the proxy instead of your network (gdirect scan warns when nearly every address answers). If TUN must stay on, route the gdirect process's transfers direct—see the Mainland China Network and Proxy Configuration Guide below.

Installation

Prebuilt binaries are provided for macOS 15+ (Apple Silicon / Intel), Linux x86_64 (glibc 2.35+, e.g. Ubuntu 22.04+, Debian 12+, Fedora, Arch), and Windows x86_64.

Option 1: One-Line Install Script

macOS / Linux:

curl -fsSL https://raw.githubusercontent.com/mizorewww/gdirect/main/scripts/install.sh | bash

Windows PowerShell:

& ([scriptblock]::Create((Invoke-RestMethod 'https://raw.githubusercontent.com/mizorewww/gdirect/main/scripts/install.ps1')))

powershell

The installer places gdirect into ~/.local/bin (%LOCALAPPDATA%\gdirect\bin on Windows). The Google GTS root certificates used for verification are embedded in the binary, so there is no CA file to install.

If you also want to use gdirect mount to mount Google Drive as a local disk, pass --with-mount (-WithMount in PowerShell) to automatically install the mount dependencies (rclone 1.75.2+, plus fuse3 on Linux or WinFsp via winget on Windows):

curl -fsSL https://raw.githubusercontent.com/mizorewww/gdirect/main/scripts/install.sh | bash -s -- --with-mount

Option 2: Homebrew (macOS / Linux)

brew tap mizorewww/gdirect https://github.com/mizorewww/gdirect
brew install mizorewww/gdirect/gdirect
brew install rclone   # Only needed for `gdirect mount`; Linux also requires `fuse3` from your system package manager

To upgrade later, run brew update && brew upgrade gdirect.

One-Time Google Cloud Setup Guide (~10 Minutes)

To avoid rate-limiting (429 Too Many Requests) or bans caused by everyone sharing a single public OAuth client, gdirect does not ship a shared client ID. Instead, you create your own Desktop app OAuth client in your personal Google Cloud project. You only need to do this once, and it is completely free.

1. Create a Project and Enable the Drive API

Open the Google Cloud Console, click the project picker at the top of the page, and click New project in the top-right corner. Name it anything you like (e.g. gdirect), leave Location as No organization for a personal account, click Create, and make sure the new project is selected at the top.
Open the Google Drive API page, confirm your new project is selected at the top, and click Enable.

Once enabled, the page shows the service name drive.googleapis.com with status Enabled.

2. Configure the OAuth App

Open Google Auth Platform and click Get started.
App information: Enter an app name (e.g. gdirect, shown on the consent screen), select your email address as the user support email, and click Next.
Audience: Choose External and click Next (personal accounts cannot choose Internal).
Contact information: Enter your email address and click Next.
Finish: Check the box agreeing to the Google API Services User Data Policy, click Continue, and then click Create.

3. Add the Drive Scope

Open Data Access in the left sidebar and click Add or remove scopes.
In the drawer that opens, select .../auth/drive ("See, edit, create, and delete all of your Google Drive files"). If you don't see it in the table, paste the URL below into Manually add scopes, click Add to table, and then click Update:
```
https://www.googleapis.com/auth/drive
```
Important: After returning to the Data Access page, make sure to click Save at the bottom of the page. Clicking Update in the drawer alone does not persist the change.

4. Create a Desktop Client and Download the JSON

Open Clients in the left sidebar and click Create client. Set Application type to Desktop app (Web application clients, API keys, and service-account JSONs will not work), give it any name (e.g. gdirect Desktop), and click Create.
In the dialog that pops up, click Download JSON before closing the modal. Once closed, Google won't let you download the secret JSON again (if you close it by mistake, just create a new client).

Newly created apps start in Testing mode, where only listed test users are allowed to sign in:

Open Audience in the left sidebar, click Add users under Test users, enter the Google account email you want to use with gdirect, and click Save.

While the app stays in Testing mode, refresh tokens expire after 7 days, requiring gdirect login --reauth once a week.

If you don't want to re-authenticate every week, simply click Publish app at the top of the Audience page to switch the status to In production. A personal app used only by you does not need to go through Google's verification review, and publishing it removes the 7-day token expiration limit.

6. Log In from the Terminal

Run the command below in your terminal, replacing the path with the JSON file you downloaded:

gdirect login --client-secret ~/Downloads/client_secret.json

Your browser will open automatically (if it doesn't, copy the URL printed in the terminal and open it in a browser on the same machine, since the OAuth flow redirects back to a temporary localhost port).
Choose your Google account. Google will warn that "Google hasn't verified this app"—this is completely normal because you just created it yourself:
- Testing mode: Click Continue (not "Back to safety").
- In production: Click Advanced in the bottom-left corner, then click Go to (unsafe).
Verify the app name and Google Drive permission scope, then click Continue.
Back in the terminal, once you see Logged in, you're all set. You can delete the downloaded JSON file now—the client credentials are saved inside local SQLite, and re-authenticating or switching accounts later only requires gdirect login --reauth.

Headless Server Tip: On a remote server without a browser, you can authenticate on your desktop first, or import an existing authorized_user credentials JSON via gdirect login --credentials <PATH>. For scripts, you can also pass an access token directly via the GDIRECT_ACCESS_TOKEN environment variable.

Mainland China Network and Proxy Configuration Guide

gdirect traffic comes in two kinds. Sign-in, account verification and token refresh (accounts.google.com, oauth2.googleapis.com, www.googleapis.com) must go through a proxy; scans, Drive API requests, uploads and downloads must connect directly to the scanned endpoints and must not be picked up by the proxy. How you configure your proxy client decides whether the two can each take their own path.

Since 0.8.0 (thanks to @Liuycer for #1), the easiest setup is rule mode + system proxy, with TUN off: sign-in requests read the system proxy settings, transfer requests bypass every application-level proxy anyway, so the two separate naturally and no direct rules are needed. That setup comes first below, followed by what goes wrong with TUN on and how to configure it.

Approach 1 (Recommended): Rule Mode + System Proxy, TUN Off

Keep your proxy client (sing-box / Mihomo / Clash, etc.) in rule mode, turn on "system proxy", and turn off TUN. Make sure accounts.google.com, oauth2.googleapis.com and www.googleapis.com go through the proxy (the google group in common rulesets already covers them). No DIRECT rules are needed.
macOS and Windows: gdirect sign-in, account verification and token refresh automatically use the manual HTTP/HTTPS system proxy. Linux, or a PAC-based system proxy: set HTTPS_PROXY for each command, pointing at your client's HTTP or mixed port (HTTP_PROXY / HTTPS_PROXY take precedence over system settings, and mount passes them on to rclone; NO_PROXY must not exclude the three domains above). Transfer commands may refresh the token too, so the simplest is to export it in your shell.
Scans and transfers are unaffected: scans, pinned API requests, uploads and downloads explicitly bypass application proxies and connect directly to the scanned endpoints. Without TUN, the proxy client never even sees these connections, so there is nothing to hijack.

gdirect login --client-secret ~/Downloads/client_secret.json   # macOS / Windows: uses the system proxy
export HTTPS_PROXY=http://127.0.0.1:7890                        # Linux / PAC: your proxy's port
gdirect scan
gdirect download FILE_ID
gdirect --google-only download FILE_ID   # Google's own front ends only, no third-party relay
# Manual overrides still work: --sni forces a TLS identity, --no-sni omits SNI (downloads only)
gdirect download FILE_ID --no-sni

gdirect scan --list shows each endpoint's latency and required TLS identity, and labels anything outside Google's address space [third-party relay]. If your ISP resets connections carrying the real SNI, there is nothing to do by hand: gdirect retries under the forged g.cn identity, and moves to the next verified endpoint when the current one is gone.

You can also quit the proxy client entirely after signing in and let gdirect transfer directly. The access token expires after about an hour, though, and refreshing it still needs the proxy, so for long transfers keep the system proxy on.

With TUN On: How Does a Proxy Hijack Direct Traffic?

With TUN virtual network interface mode on, every packet on the machine passes through the proxy's routing engine first, and gdirect bypassing proxies at the application layer doesn't help. Let's trace what happens:

The destination is one of Google's own overseas front ends: At the socket layer, gdirect swaps the destination for a scanned endpoint—usually a front end inside Google's own address space (such as 142.250.x.x or 192.178.x.x), or, by default, the relay 183.56.143.147 it tries first. So that the Google front end routes the request by hostname and local certificate verification succeeds, the TLS handshake (ClientHello SNI) and HTTP Host header still carry www.googleapis.com, drive.usercontent.google.com, and *.googleusercontent.com (the SNI is g.cn under the forged identity).
IP rules: Google's addresses go to the proxy by default: Common rule templates contain IP rules such as GEOIP,google or RULE-SET,google-ip that send Google's ranges to the proxy. That is right for a browser (most Google IPs really are unreachable in China), but it also sends the front ends gdirect worked to find—the ones that happen to be reachable—into the proxy.
Domain rules: TUN sniffing (sniff) hijacks the connection: When gdirect connects to port 443 on a front end, the TUN interface intercepts the packet and uses sniff (TLS SNI inspection) to read drive.usercontent.google.com or www.googleapis.com. In rulesets such as DustinWin/ruleset_geodata, those domains belong to the proxied google set, not the directly reachable google-cn set (which lists domains like dl.google.com, g.cn, and fonts.googleapis.com):
- rule_set: google-cn (DIRECT) $\rightarrow$ Miss;
- rule_set: google (PROXY) $\rightarrow$ Match!
- Even when an IP rule would let the destination through (the relay 183.56.143.147 is a Chinese IP and matches GEOIP,CN, for instance), domain rules come before IP rules in mainstream templates, so the sniffed Drive hostname pulls the connection back into the proxy anyway. Sign-in needs www.googleapis.com proxied, and transfers use that very same hostname: that is why the two kinds of traffic are hard to separate under TUN.
TUN "answers" TCP for every address: In TUN mode, the proxy accepts each connection locally and then dials the destination itself. The first step of gdirect scan, a TCP probe, therefore "succeeds" almost everywhere, and the scan measures the proxy rather than your network. When over 90% of a batch answers, scan prints a warning asking you to route Google IPs DIRECT and scan again.

Approach 2: When TUN Must Stay On, Keep Transfers Away from Sniffing and IP Rules

If TUN has to stay on around the clock, the goal is: sign-in requests keep going through the proxy by domain, while gdirect's transfer connections go direct. There are two ways:

Route by process (simplest on desktop): before GEOSITE,google, GEOIP,google and similar rules, put the proxy rules for sign-in and token refresh, then PROCESS-NAME,gdirect,DIRECT. The catch: the www.googleapis.com account check during gdirect login also matches the process rule, goes direct and fails, so for the moment of signing in switch back to Approach 1 (TUN off, system proxy on) or disable the process rule temporarily. Sign-in is rare (once, for an app in production); token refresh afterwards only uses oauth2.googleapis.com, which sits above the process rule and stays proxied.
Skip sniffing for the scanned IPs: exclude the endpoints listed by gdirect scan --list in your client's sniffing settings, and route those IPs (or all of Google's ranges) direct. Transfer connections then carry no domain, and the www.googleapis.com proxy rule only matches sign-in. The downsides: endpoints change with each scan, and routing Google's ranges direct breaks Google services in your browser.

Note: A regular browser visiting www.googleapis.com connects to the blackholed IP that DNS returns, not to a scanned front end the way gdirect does. So don't route the Drive domains or Google's IP ranges DIRECT wholesale for gdirect; that breaks every web page and Google service in your browser that depends on them. fake-ip doesn't affect transfers either: gdirect connects to IPs without DNS, and the only effect is that 198.18.x.x fake answers seen during scan are dropped, leaving a few fewer candidates.

For Mihomo (Clash), routing by process means adding the following before your GEOSITE,google, GEOIP,google and similar rules:

rules:
  # 1. Sign-in and token refresh go through the proxy (must come before the process rule)
  - DOMAIN,accounts.google.com,PROXY
  - DOMAIN,oauth2.googleapis.com,PROXY

  # 2. Everything else from the gdirect process (scan, API, download, upload) goes DIRECT,
  #    so sniffed domain rules and Google IP rules don't hijack it.
  #    For gdirect login, turn TUN off and use the system proxy, or disable this rule
  - PROCESS-NAME,gdirect,DIRECT

yaml

If you use sing-box, likewise put a proxy rule for oauth2.googleapis.com and then a process_name: gdirect direct rule before rule_set: ["google"] and any Google IP rules in route.rules.

Daily Usage & Troubleshooting Reference

Common Commands

gdirect                                   # Same as `gdirect browse`, opens the interactive TUI
gdirect browse -o ~/Downloads             # Set download directory (defaults to ./downloads)
gdirect ls root --json                    # List directory contents (supports JSON output)
gdirect download <FILE_ID_OR_SHARE_URL>   # Download a file; re-run the same command to resume after interruption
gdirect download FILE_ID -o file.zip
gdirect download DOC_ID --export application/pdf -o doc.pdf
gdirect upload ./file.zip                 # 32 MiB chunked upload with automatic resume
gdirect upload ./file.zip --parent FOLDER_ID --name backup.zip
gdirect mount                             # Mount at ~/GoogleDrive (uses native NFS on macOS, no FUSE needed)
gdirect unmount                           # Unmount and clean up temporary exported credentials
gdirect scan                              # Scan again; scan --list shows the results

Since 0.6.0, several Google accounts can share one OAuth client:

gdirect login --add                       # Sign in another account in the browser
gdirect accounts                          # List accounts; * marks the default
gdirect switch work                       # Make work@… the default
gdirect -a me@gmail.com browse            # Use one account for a single command (or set GDIRECT_ACCOUNT)

The browse TUI uses a yazi-style three-column layout showing the parent folder, the current folder and a preview side by side, with multi-select, whole-folder downloads and file operations in Drive. Press ? for every key:

Key	Action
`j / k`, `↑ / ↓`	Select
`g / G`, Home/End	First / last item
`Ctrl-d / Ctrl-u`, PgDn/PgUp	Half a page / a page down or up
Enter, `l / →`	Open folder
`h / ←`, Backspace	Parent folder
`/`	Filter the current folder by name. Enter keeps the filter, Esc clears it
`Ctrl-r`, F5	Refresh
Space	Mark or unmark an item; marks are kept when you change folders
`Ctrl-a`	Mark everything shown (press again to unmark)
Esc	Clear the filter, then the marks, then the clipboard
`a`	Create an empty file; a name ending in `/` creates a folder
`r`	Rename
`y` / `x`	Copy / cut the marked items (or the selected one)
`p`	Paste into the current folder. Cut items are moved; copied files are duplicated
`D`, Delete	Move the marked items to the Drive trash (asks first)
`u`	Upload a local file into the current folder
`d`	Queue the marked items for download. A folder is downloaded with all its subfolders; Google Docs, Sheets and Slides are exported as PDF
`+ / -`	Connections for the next download (`1–16`), saved as your default
`c`	Cancel the current download and keep the partial data
`R`	Retry failed or cancelled downloads
`X`	Remove downloads that have not started yet
`q`	Quit

Downloads run one at a time; the file list shows each file's state (queued, downloading, saved, failed), and selecting a failed file shows the full error.

For tricky network environments, there are transport-layer overrides:

--google-only (global; or GDIRECT_GOOGLE_ONLY=1): use no third-party relay; scans and transfers use only endpoints inside Google's published address space;
--ip IP: force a specific IP, skipping scan results and failover;
--sni HOST / --no-sni: force a TLS identity, or omit SNI altogether (--sni also works for ls and upload; --no-sni is download-only). Without them, gdirect uses the scanned identity and falls back automatically;
--ca-cert FILE: use this CA bundle instead of the embedded Google GTS roots;
-j N: set concurrent connections (1–16) for this command only;
gdirect scan --ips FILE: probe a custom candidate list (one public IP per line, # for comments). Addresses there are not limited to Google's space, but those outside it are labelled [third-party relay], and --google-only keeps transfers off them.

Troubleshooting Quick Reference

Symptom	Fix
`access_denied` or message that the app is being tested	Add your Google account as a Test user, or publish the app (Step 5 of the setup guide)
`redirect_uri_mismatch`	The downloaded JSON is not from a Desktop app client; recreate it following Step 4
`Drive API 403 … insufficient permission`	Enable the Google Drive API in the same project as your OAuth client (Step 1), save the Drive scope (click Save at the bottom in Step 3), then run `gdirect login --reauth`
`refresh credential expired or revoked`	Run `gdirect login --reauth`. If this happens every 7 days, publish the app from Testing to In production (Step 5)
`scan` warns `almost every address answered TCP`	A TUN proxy is answering for every address, so the scan measured the proxy. Turn TUN off as described above, or route the `gdirect` process / Google IPs `DIRECT`, then scan again
`no usable Google front end found`	No usable endpoint was found from this network. Rule out a TUN proxy first; otherwise scan again later (reachable front ends change), or supply candidates with `gdirect scan --ips FILE`
`libaria2 code 1` mid-download (connection or TLS failure)	The real SNI was reset or the endpoint died. `gdirect` retries under `g.cn`, then resumes on the next verified endpoint and marks the dead one; it only fails once every endpoint has, and re-running the command rescans. `--sni g.cn` / `--no-sni` still work by hand
`cannot verify the account` or `cannot reach Google's token endpoint` during sign-in	Sign-in, account verification or token refresh didn't go through the proxy. Make sure your client's system proxy is on and `accounts.google.com`, `oauth2.googleapis.com` and `www.googleapis.com` are proxied; set `HTTPS_PROXY` on Linux or with PAC; with TUN and a process rule, turn TUN off for the moment of signing in (see Approach 2)
An IP in `scan --list` shows `[third-party relay]`	The address is outside Google's published space: a third-party relay, such as `183.56.143.147`, which is tried first by default. It cannot read your traffic but sees your IP; use `--google-only` if that matters to you

Correction and Improvements: 183.56.143.147 Is Not Google

After the first version of this post went out, a reader, Ken, emailed to point out several factual errors: 183.56.143.147 is not a Google server; the google-cn nodes inside China will not fetch Drive for you; and "GFE" is not a kind of IP or server. We spent a day checking each point, and he was right on all of them. This section lays out how we checked, what changed, and where the new approach falls short. Thank you, Ken, for reading carefully and taking the time to write.

What the First Version Got Wrong

The first version claimed that 183.56.143.147 was a Google google-cn edge node on China Telecom in Guangzhou; that these domestic nodes ran Google's front end (GFE) and reached Drive's backends over Google's internal network; and that connecting to it therefore kept traffic from "physically crossing the firewall". The scan also used it as its built-in seed.

The IP came from the Cealing-Host list used by Sheas-Cealer, where every Google entry points at the same address. It presents Google's genuine certificate and the Drive API answers normally through it; we took that as proof it was Google and never checked who actually runs it.

Check 1: Who Does Google See as the Client?

Google has a diagnostic endpoint, https://redirector.googlevideo.com/report_mapping?di=no, that reports the client IP it sees. Testing from a China Unicom line in Shanghai (bound to the physical interface to bypass the local VPN):

A direct request: Google reports our own IP;
Through 183.56.143.147: Google reports 122.10.103.81, an address in Hong Kong (AS136897, Enjoyvc).

So the connection enters Google from Hong Kong. whois shows 183.56.143.147 belongs to CHINANET Guangdong, and neither it nor 122.10.103.81 is in Google's published ranges (goog.json). Conclusion: it is a third-party TCP relay (reverse proxy) that forwards traffic to Hong Kong and on to Google.

It doesn't touch TLS; it just forwards the byte stream, so the client sees Google's genuine GTS certificate and gdirect verifies it against Google's GTS roots as usual. That means:

The relay cannot see: access tokens, file contents or request paths, all of which stay inside end-to-end TLS; nor can it forge a certificate to intercept.
The relay can see: your IP, traffic volume and timing; and, unless the forged identity is used, the hostname in the TLS SNI (e.g. www.googleapis.com). Under the g.cn identity, it only sees a connection to g.cn.

Check 2: Can the Genuine Google Nodes in China Serve Drive?

Google does have nodes in China, just not the kind the first version described. We tested two groups:

203.208.40.x / 203.208.41.x, registered to GOOGLECN (Beijing Gu Xiang, Google's China entity);
Google cache nodes inside ISP networks, such as 211.95.34.x inside China Unicom Shanghai, which domestic DNS returns for dl.google.com, fonts.googleapis.com, g.cn and www.gstatic.com.

Both accept TCP and TLS and present Google certificates. Requesting each service by Host header makes the picture clear:

Requested host	Result
`www.google.cn`, `fonts.googleapis.com`	200
`www.gstatic.com/ipranges/goog.json`	200
`www.googleapis.com/drive/v3/…`	404
`oauth2.googleapis.com`	404
`drive.usercontent.google.com`	404

They serve a whitelist, and Drive is not on it. The path "domestic node, then Google's internal network, then Drive" does not exist. (They do serve Google's IP range lists, which comes in handy later.)

Check 3: Other Intuitive Candidates

The Google IPs domestic DNS returns for www.googleapis.com and drive.usercontent.google.com (74.125.x, 172.217.x, 142.250.157.x and so on) are real, but TCP times out: they are blackholed.
DNS for drive.google.com is poisoned; the answers (69.63.184.30, 128.242.240.59) are not Google at all.
Firebase Hosting and Google Cloud load-balancer IPs present their customers' certificates and cannot be used for Google APIs.
One Cloud Storage IP (142.250.73.123) was reachable, but it is a GCS-only front end that answers every Host as GCS.

Check 4: Sweeping Google's Own Front Ends

If every IP that DNS hands out is blocked, what about the ones it doesn't? Google publishes all of its ranges (goog.json) and the part leased to Cloud customers (cloud.json); the difference is Google's own service space. We swept the blocks where front ends concentrate (142.250.0.0/15, 172.217.0.0/16, 172.253.0.0/16, 173.194.0.0/16, 74.125.0.0/16, 192.178.0.0/15, 64.233.160.0/19, 66.102.0.0/20, 108.177.0.0/17, 209.85.128.0/17, 216.58.192.0/19, 216.239.32.0/19, 108.170.192.0/18, 72.14.192.0/18): about 2,500 /24s, 12 host numbers each (.4 .14 .91 .100 .113 .123 .132 .138 .155 .187 .207 .219). For each address: TCP 443 first; for those that answered, a full check under both the real SNI and g.cn—an unauthenticated Drive API request had to return the expected 401 JSON, and a public test file had to download correctly.

Sweep of Google's own front endsChina Unicom Shanghai · 2026-10-26

Funnel~2,500 /24s × 12 host numbers

Addresses probed: 30528
Answered TCP 443: 1423; 4.7%
Fully working: +104; Drive API + test download passed
Spread: ~65 /24s; clustered; 192.179.50/60/61 had 7 each

Performancefrom the same Unicom line

TCP connect: 268 ms; p50; p90 was 511 ms
Single-connection download: 4–6 MB/s; 27 MB file, Host dl.google.com

Every working address passed under both the real SNI and the forged g.cn identity.

The 1,319 addresses that answered TCP but failed

Reason	Count
Certificate mismatch	~1030
TLS timeout or reset	~290

Certificate mismatches were mostly Cloud Storage-only front ends; TLS timeouts and resets may be SNI-based interference.

The host numbers show a clear pattern: .187, .207 and .219 never worked (all Cloud Storage-only front ends), while .100, .113, .155, .132, .138 and .91 worked best.

Are they really Google, and really direct? Three pieces of evidence: every address is in goog.json; whois says Google LLC; and traceroute from the Unicom line goes straight into Google's backbone (142.251.x, 192.178.x, 172.253.x) with no third party in between.

Why aren't they blocked? Our guess: domestic DNS never hands them out, so they haven't made it onto the blocklist. That also means they may be blocked as soon as usage grows—which is exactly why gdirect discovers them dynamically instead of shipping a fixed list.

A Note on Terminology: What GFE Is

The first version used "GFE" as if it were a category of node, which is inaccurate. Google's servers come in kinds such as GWS (Google Web Server), GVS (video servers) and GGC (Google Global Cache nodes inside ISP networks); the Google Front End (GFE) is a component of Google's infrastructure that the Google security whitepaper describes as terminating TLS, providing DDoS protection and load balancing before handing requests to backend services. This post now uses "GFE" only in that sense: Google's front-end serving layer, which routes requests by Host. It is not an IP category. "Google front end" in this post means a server in Google's own address space served by that layer.

What Changed in gdirect 0.7.0 / 0.8.0

Google's address space, embedded and refreshed weekly (0.7.0; src/ranges.rs, src/google-ranges.txt, scripts/update-ranges.py): a snapshot of goog.json minus cloud.json ships in the binary and is refreshed weekly at scan time. The refresh goes over the system network first—in China, www.gstatic.com resolves to exactly the kind of in-ISP Google cache above, which serves the range lists—and through a verified endpoint otherwise. Lists that look truncated are rejected.
The scan became a front-end sweep (0.7.0; src/scan.rs): candidates come only from inside Google's space: earlier verified endpoints, DNS answers (poisoned ones fall outside and are dropped), the /24s where our sweep found working front ends (src/frontend-seeds.txt), then the core front-end blocks, then everything else, each part shuffled per user. Only the productive host numbers are tried. Each batch of 2,048 addresses gets a TCP probe, then full verification of what answered; when one works, the rest of its /24 goes next; the sweep stops at 8 verified endpoints. It warns when nearly everything answers TCP (a TUN proxy intercepting).
Failover (0.7.0; Api::deliver and downloads): when the current endpoint stops answering, gdirect tries the other TLS identity, then each other endpoint verified within the last day; downloads resume on the new endpoint.
The relay: 0.7.0 dropped 183.56.143.147 from the scan and purged it from existing databases once. In practice the relay is fast and stable, so 0.8.0 tries it first by default and falls back to the scanned Google front ends when it fails, and the 0.7.0 purge is gone. It also adds a global --google-only option (environment variable GDIRECT_GOOGLE_ONLY=1): the scan skips relays and transfers use only endpoints inside Google's space. Either way, endpoints outside Google's space are labelled [third-party relay] in scan and scan --list.
Sign-in through the system proxy (0.8.0, #1 contributed by @Liuycer): sign-in, account verification and token refresh use the macOS / Windows HTTP/HTTPS system proxy (and HTTPS_PROXY), so a rule-mode proxy client with TUN off is enough to sign in; scans, pinned API requests, uploads and downloads still connect directly.

Which to choose: the relay cannot read your tokens or files, but its operator—whom we don't know—sees your IP, traffic volume and timing, and (without the forged identity) which Google host you connect to. If that bothers you, use --google-only; you can also force the forged identity with --sni g.cn so the relay doesn't even see the hostname.

Why We Don't Bundle Reverse Proxies Found Online

Search engines for internet-facing hosts such as fofa turn up plenty of reverse proxies presenting Google certificates, and someone suggested bundling a batch of them. We didn't: most are someone else's misconfigured servers, accidentally left open, and using them en masse without permission is freeloading on other people's machines and bandwidth; their operators see every user's IP and access times; and such servers tend to be shut down or blocked quickly, so a bundled list goes stale fast. 183.56.143.147 is a relay a public list has used for a long time; we label it clearly as third-party, try it first by default, and let you turn it off with --google-only. Beyond that, gdirect doesn't seek out any third-party server. Candidates you find yourself can still be tested with gdirect scan --ips FILE, and anything outside Google's space is labelled the same way.

Limits

All of the results above come from one line (China Unicom Shanghai), one location, one day (2026-10-26). China Telecom, China Mobile and other regions are unverified; the reachable front ends there may be entirely different, or there may be none.
These front ends are probably reachable only because nobody has noticed them yet. They may be blocked at any time; when that happens gdirect can only rescan for others, and if it finds none, fall back to the relay or a proxy.
4–6 MB/s per connection with a p50 connect time of 268 ms is what you'd expect from a direct cross-border link, not "saturating your broadband". Multiple connections help, depending on the line.

If you test from another ISP or region, I'd love to hear the results, whether or not it works.

Technical Deep Dive: How Is It Implemented Under the Hood?

If you're curious about how gdirect combines Rust with a custom-patched libaria2 build to pull this off, here is a walkthrough of the source code.

1. Where Candidates Come From: Google's Own Address Space and the Sweep Order

When a transfer command runs without an endpoint verified in the last 24 hours in local SQLite, src/scan.rs starts a scan. By default 0.8.0 probes the known relay 183.56.143.147 first (skipped under --google-only), then sweeps Google's front ends anyway, as backups for when the relay fails:

src/scan.rs0.8.0: verify the known relay before sweeping, unless --google-onlyL336–346 · 1099f77 ↗

336        if !google_only {
337            let relays: Vec<IpAddr> = RELAYS.iter().filter_map(|r| r.parse().ok()).collect();
338            eprintln!("Checking known relays (not Google; use --google-only to skip)…");
339            record(
340                db,
341                &google,
342                verify(&relays, usize::MAX),
343                |_| true,
344                &mut warned,
345            )?;
346        }

There is exactly one relay, hard-coded in RELAYS with a comment stating it is not Google. For transfers, usable sets the order of endpoints: the relay first, the rest by latency; --google-only drops every endpoint outside Google's space, including ones you added by hand with scan --ips:

src/scan.rs0.8.0: the known relay and endpoint orderingL32–54 · 1099f77 ↗

32/// Known third-party relays that forward to Google. Not Google: the operator
33/// sees client IPs and, without the forged identity, the hostname, but TLS
34/// stays end to end and is verified against Google's roots. They are fast and
35/// stable in mainland China, so they are tried first unless --google-only.
36const RELAYS: [&str; 1] = ["183.56.143.147"];
37fn relay(ip: IpAddr) -> bool {
38    RELAYS.iter().any(|r| r.parse() == Ok(ip))
39}
40/// Endpoints in the order transfers should use them: known relays first
41/// (unless `google_only`, which drops everything outside Google's space),
42/// then the rest as given (fastest first).
43pub fn usable(
44    routes: Vec<crate::store::Route>,
45    google: &Ranges,
46    google_only: bool,
47) -> Vec<crate::store::Route> {
48    let mut routes: Vec<_> = routes
49        .into_iter()
50        .filter(|r| !google_only || google.contains(r.ip))
51        .collect();
52    routes.sort_by_key(|r| !relay(r.ip));
53    routes
54}

src/main.rs0.8.0: pick the current endpoint and backups; a pinned --ip gets no failoverL194–220 · 1099f77 ↗

194fn direct_route(db: &store::Store, route: &Route, google_only: bool) -> Result<Direct> {
195    let explicit = route.sni();
196    let auto = explicit.is_none();
197    if let Some(ip) = route.ip {
198        return Ok(Direct {
199            ip,
200            sni: explicit,
201            auto,
202            backups: Vec::new(),
203        });
204    }
205    let google = ranges::current(db);
206    let mut routes = scan::usable(db.routes(16)?, &google, google_only);
207    if routes.is_empty() {
208        scan::run(db, None, google_only)?;
209        routes = scan::usable(db.routes(16)?, &google, google_only);
210    }
211    let identity = |sni: String| (!sni.is_empty()).then_some(sni);
212    let mut routes = routes.into_iter().map(|r| (r.ip, identity(r.sni)));
213    let (ip, stored) = routes.next().context("no verified endpoint")?;
214    Ok(Direct {
215        ip,
216        sni: explicit.or(stored),
217        auto,
218        backups: routes.take(8).collect(),
219    })
220}

The front-end sweep code below is from 0.7.0; it is unchanged in 0.8.0.

The sweep is bounded by Google's own address space in src/ranges.rs: goog.json lists every address Google owns, cloud.json the part that can be assigned to Google Cloud customers, and the difference is where Google's own services live. A snapshot is embedded and refreshed weekly from www.gstatic.com; a list that comes back implausibly small (truncated or tampered with) is rejected in favour of the old one:

src/ranges.rsGoogle's own space: goog.json minus cloud.jsonL128–150 · 338cd96 ↗

128    /// Google's own space from the published feeds: everything Google owns
129    /// minus what Google Cloud customers can be assigned.
130    pub fn from_feeds(goog: &str, cloud: &str) -> Result<Self> {
131        let list = |json: &str| -> Result<Self> {
132            let data: serde_json::Value = serde_json::from_str(json)?;
133            let prefixes = data["prefixes"]
134                .as_array()
135                .context("IP range feed has no prefixes")?;
136            Self::from_prefixes(
137                prefixes
138                    .iter()
139                    .filter_map(|p| p["ipv4Prefix"].as_str().or(p["ipv6Prefix"].as_str())),
140            )
141        };
142        let (goog, cloud) = (list(goog)?, list(cloud)?);
143        let ranges = Self {
144            v4: subtract(&goog.v4, &cloud.v4),
145            v6: subtract(&goog.v6, &cloud.v6),
146        };
147        // A truncated or hostile feed must not shrink the trusted space to nothing.
148        ensure!(ranges.v4.len() >= 16, "IP range feed looks incomplete");
149        Ok(ranges)
150    }

Candidates are queued by priority, and only addresses inside Google's space are accepted: earlier verified endpoints and DNS answers first (poisoned answers fall outside the space and are dropped), then the /24s where our sweep found working front ends (src/frontend-seeds.txt, 67 of them), then the core front-end blocks, and finally everything else. Each part is shuffled per user so everyone doesn't pile onto the same few front ends:

src/scan.rsSweep order: inside Google's space only, by priorityL96–131 · 338cd96 ↗

96/// Sweep order: earlier verified endpoints, DNS answers, seed /24s, then
97/// Google's core front-end blocks and finally the rest of its space, each
98/// shuffled. Only addresses inside `google` are ever considered.
99fn plan(
100    google: &Ranges,
101    known: &[IpAddr],
102    dns: &[IpAddr],
103    shuffle: &mut Shuffle,
104) -> VecDeque<IpAddr> {
105    let core = Ranges::parse(&CORE.replace(' ', "\n")).expect("core blocks must parse");
106    let seeds = Ranges::parse(SEEDS).expect("seed list must parse");
107    let mut seen = HashSet::new();
108    let mut order = VecDeque::new();
109    let mut add = |ip: IpAddr, order: &mut VecDeque<IpAddr>| {
110        if public(ip) && google.contains(ip) && seen.insert(ip) {
111            order.push_back(ip);
112        }
113    };
114    for &ip in known.iter().chain(dns) {
115        add(ip, &mut order);
116    }
117    let mut seed_nets: Vec<u32> = seeds.slash24s().collect();
118    shuffle.apply(&mut seed_nets);
119    let (mut core_nets, mut rest_nets): (Vec<u32>, Vec<u32>) = google
120        .slash24s()
121        .filter(|n| !seeds.contains(IpAddr::V4(Ipv4Addr::from(*n))))
122        .partition(|n| core.contains(IpAddr::V4(Ipv4Addr::from(*n))));
123    shuffle.apply(&mut core_nets);
124    shuffle.apply(&mut rest_nets);
125    for net in seed_nets.into_iter().chain(core_nets).chain(rest_nets) {
126        for ip in hosts(net) {
127            add(ip, &mut order);
128        }
129    }
130    order
131}

Only 7 host numbers per /24 are tried, based on the statistics from the sweep above: .187, .207 and .219 never worked (all Cloud Storage-only front ends), so they are skipped:

src/scan.rsSweep parameters: target, batch size, host numbers and core blocksL16–31 · 338cd96 ↗

16const LIMIT: usize = 256;
17/// Verified endpoints to collect before a sweep stops.
18const TARGET: usize = 8;
19/// Upper bound on TCP probes in one sweep.
20const MAX_PROBES: usize = 60_000;
21/// Addresses swept per batch: TCP first, then full verification of what answered.
22const BATCH: usize = 2048;
23const TCP_WORKERS: usize = 256;
24const TCP_TIMEOUT: Duration = Duration::from_secs(2);
25/// Host numbers inside a /24 where Google's general front ends answer. .187,
26/// .207 and .219 are Cloud Storage-only front ends and never serve Drive.
27const OFFSETS: [u8; 7] = [100, 113, 155, 132, 138, 91, 123];
28/// Blocks that hold most of Google's general front ends; swept before the
29/// rest of Google's space.
30const CORE: &str = "142.250.0.0/15 172.217.0.0/16 172.253.0.0/16 173.194.0.0/16 74.125.0.0/16 192.178.0.0/15 64.233.160.0/19 66.102.0.0/20 108.177.0.0/17 209.85.128.0/17 216.58.192.0/19 216.239.32.0/19 108.170.192.0/18 72.14.192.0/18";
31const SEEDS: &str = include_str!("frontend-seeds.txt");

Before probing, public(ip) filters out private, loopback, link-local and 198.18.0.0/15 (the default Clash / Mihomo fake-ip range) addresses.

2. A Two-Stage Sweep: TCP First, Then Full Verification

Most Google addresses don't even accept TCP from China, so running full TLS + API verification on every one would be far too slow. The sweep runs in two stages: each batch of 2,048 addresses gets a TCP connect with a 2-second timeout from 256 threads, and only the addresses that answer go on to full verification by 8 threads. When an address verifies, the other host numbers in its /24 jump to the front of the queue, since working front ends tend to come in clusters. The sweep stops at 8 verified endpoints:

src/scan.rsSweep loop: TCP probe, full verification, neighbours firstL321–352 · 338cd96 ↗

321        while good < TARGET && probed < MAX_PROBES && !queue.is_empty() {
322            let batch: Vec<IpAddr> = std::iter::from_fn(|| queue.pop_front())
323                .filter(|ip| tried.insert(*ip))
324                .take(BATCH)
325                .collect();
326            probed += batch.len();
327            let answered = tcp_sweep(&batch);
328            open += answered.len();
329            // Real networks block most of Google; a TUN proxy accepts every connect.
330            if batch.len() >= 256 && answered.len() * 10 >= batch.len() * 9 && !intercepted {
331                intercepted = true;
332                eprintln!(
333                    "warning: almost every address answered TCP, so a TUN proxy or VPN is probably intercepting; results reflect the proxy, not your direct network. Route Google IPs DIRECT for a real scan"
334                );
335            }
336            let candidates: Vec<IpAddr> = answered.into_iter().map(|(ip, _)| ip).collect();
337            reached.extend(&candidates);
338            let verdicts = verify(&candidates, TARGET - good);
339            let working = record(db, &google, verdicts, |ip| known.contains(&ip), &mut warned)?;
340            good += working.len();
341            // Front ends come in groups: the rest of a working /24 goes next.
342            for ip in working.iter().rev() {
343                if let IpAddr::V4(v4) = ip {
344                    for neighbour in hosts(u32::from(*v4) & !0xff) {
345                        if !tried.contains(&neighbour) {
346                            queue.push_front(neighbour);
347                        }
348                    }
349                }
350            }
351            eprintln!("  … probed {probed}, {open} answered TCP, {good} verified");
352        }

Note the TUN warning in the loop: a real network blocks the vast majority of Google's addresses, so if more than 90% of a batch answers, a TUN proxy is almost certainly accepting every connection locally.

Full verification probes each address at most twice: the real SNI first, then the forged g.cn identity:

src/scan.rsFull verification: real SNI first, forged identity on failureL172–178 · 338cd96 ↗

172                    let (result, sni) = match probe(ip, false) {
173                        Ok(ok) => (Ok(ok), ""),
174                        Err(real) => match probe(ip, true) {
175                            Ok(ok) => (Ok(ok), crate::api::FORGED_HOST),
176                            Err(forged) => (Err(format!("{real:#} | forged: {forged:#}")), ""),
177                        },
178                    };

src/scan.rsOne probe: TLS certificate + unauthenticated Drive API + test fileL189–243 · 338cd96 ↗

189// A missing or changed public test file is reported as a warning, not as a dead IP.
190//
191// `forged` selects the fallback identity: the URL (and therefore SNI and
192// certificate name) is g.cn while the Host header still carries the real
193// Google host. Google's front end routes by Host, so the same probes work.
194fn probe(ip: IpAddr, forged: bool) -> Result<(i64, Option<String>)> {
195    use crate::api::{API_HOST, FORGED_HOST};
196    const FILE_HOST: &str = "drive.usercontent.google.com";
197    let (api_url, file_url) = if forged {
198        (FORGED_HOST, FORGED_HOST)
199    } else {
200        (API_HOST, FILE_HOST)
201    };
202    let client = crate::api::client_builder()
203        .no_proxy()
204        .redirect(reqwest::redirect::Policy::none())
205        .connect_timeout(Duration::from_secs(3))
206        .timeout(Duration::from_secs(5))
207        .resolve(api_url, SocketAddr::new(ip, 443))
208        .resolve(file_url, SocketAddr::new(ip, 443))
209        .build()?;
210    let start = Instant::now();
211    let response = client
212        .get(format!("https://{api_url}/drive/v3/about?fields=user"))
213        .header(reqwest::header::HOST, API_HOST)
214        .send()
215        .map_err(|e| e.without_url())?;
216    ensure!(
217        response.status() == reqwest::StatusCode::UNAUTHORIZED,
218        "unexpected Drive API status {}",
219        response.status()
220    );
221    let data: serde_json::Value = response.json()?;
222    ensure!(data["error"]["code"] == 401, "invalid Drive API response");
223    let response = client
224        .get(format!(
225            "https://{file_url}/download?id=0B9P1L--7Wd2vU3VUVlFnbTgtS2c&export=download&confirm=t"
226        ))
227        .header(reqwest::header::HOST, FILE_HOST)
228        .send()
229        .map_err(|e| e.without_url())?;
230    let status = response.status();
231    let warning = if matches!(
232        status,
233        reqwest::StatusCode::FORBIDDEN | reqwest::StatusCode::NOT_FOUND
234    ) {
235        Some(format!("HTTP {status}"))
236    } else {
237        ensure!(status.is_success(), "Drive download probe HTTP {status}");
238        let mut bytes = Vec::new();
239        response.take(32).read_to_end(&mut bytes)?;
240        (bytes != b"spam\n").then(|| "unexpected content".into())
241    };
242    Ok((start.elapsed().as_millis() as i64, warning))
243}

A few design choices stand out:

.no_proxy() + .resolve(): bypasses environment proxy variables and binds the connection to ip:443 inside the HTTP client, while enforcing full certificate verification against the embedded GTS roots (ruling out ISP captive portals, Cloud load balancers with customer certificates, and MITM boxes).
Credential-less API verification: calls GET /drive/v3/about?fields=user without any user token. A front end that really serves the Drive API always replies with HTTP 401 Unauthorized and JSON error.code == 401; the whitelist-only Google nodes in China return 404, and Cloud Storage-only front ends fail certificate verification. This confirms application-layer routing without ever exposing the user's Bearer token to an unknown IP.
Download path probe: next it requests a tiny public test file. If Google rate-limits the test file with HTTP 403/404, the IP is still accepted with a warning as long as the Drive API check passed, so a throttled test file doesn't disqualify every good IP.
Dual-identity probing: under the forged identity, the URL, TLS SNI and certificate name all become g.cn, while .header(HOST, API_HOST) sets the Host header back to the real hostname. GFE routes by Host, so the same API and test-file probes verify the forged path end to end. The identity that passes is stored in the sni column of the endpoints table, and gdirect scan --list shows which identity each IP needs.
Ownership labels: when printing results, the scan checks again whether each address is inside Google's space and labels the rest [third-party relay]. The only out-of-space address a default scan touches is the built-in relay 183.56.143.147; gdirect scan --ips FILE probes whatever you give it and labels it the same way.

Want to supply your own candidates? Save IPs one per line in ips.txt and run gdirect scan --ips ips.txt; the scanner verifies the certificate, the Drive API and the test download for each. Note that the in-ISP Google nodes domestic DNS returns for google-cn domains like dl.google.com and g.cn do not serve Drive and will all fail. Front-end addresses inside Google's space that DNS normally never hands out are a better bet.

3. Forged-Identity Fallback and Endpoint Failover

The scan answers "which IP to connect to", but real networks add two more variables: which identity to connect as, and whether that IP still works tomorrow. Some censoring middleboxes don't block the IP itself but reset TLS connections whose ClientHello carries a real Google hostname (SNI). That is the classic cause of "the connection suddenly dies mid-download", and changing IPs alone doesn't fix it. And the front ends we found may be blocked at any time.

gdirect wraps every pinned-IP request path in the deliver function in src/api.rs. On a connect error or timeout it first retries under the other TLS identity on the current endpoint; if that fails too, it marks the endpoint dead in the database and moves through the other endpoints verified within the last day (each tried with its stored identity first, then the other one). Whatever works is kept for the rest of the process, and the working identity is persisted in that IP's sni column:

src/api.rsdeliver: switch identity, then switch endpointL145–218 · 338cd96 ↗

145    /// Sends one request. On a connect/timeout failure it is retried under
146    /// the other TLS identity (real <-> forged g.cn), then on each backup
147    /// endpoint in turn; dead endpoints are marked in SQLite. Whatever works
148    /// is kept for this process and the working identity is persisted.
149    pub(crate) fn deliver(
150        &self,
151        build: impl Fn(&Client, Option<&str>) -> reqwest::blocking::RequestBuilder,
152    ) -> Result<reqwest::blocking::Response> {
153        let (client, ip, forged) = {
154            let live = self.live.lock().unwrap();
155            (live.client.clone(), live.ip, live.forged.clone())
156        };
157        let send = |client: &Client, forged: Option<&str>| {
158            build(client, forged)
159                .header(reqwest::header::HOST, API_HOST)
160                .send()
161                .map_err(|e| e.without_url())
162        };
163        let transport = |e: &reqwest::Error| e.is_connect() || e.is_timeout();
164        let first = match send(&client, forged.as_deref()) {
165            Err(error) if ip.is_some() && transport(&error) => error,
166            result => return Ok(result?),
167        };
168        let db = Store::open(&self.db).ok();
169        let mut endpoint = ip.map(|ip| (ip, forged.clone()));
170        let mut tried_current = true;
171        let mut last = first;
172        while let Some((ip, preferred)) = endpoint {
173            // The identity already tried on the current endpoint is skipped.
174            let mut identities = Vec::new();
175            if !tried_current {
176                identities.push(preferred.clone());
177            }
178            if self.auto {
179                identities.push(match &preferred {
180                    Some(_) => None,
181                    None => Some(FORGED_HOST.to_owned()),
182                });
183            }
184            for identity in identities {
185                let client = client_for(Some(ip), identity.as_deref())?;
186                match send(&client, identity.as_deref()) {
187                    Ok(response) => {
188                        let mut live = self.live.lock().unwrap();
189                        live.client = client;
190                        live.ip = Some(ip);
191                        live.forged = identity.clone();
192                        if let Some(db) = &db {
193                            let _ = db
194                                .set_sni(&ip.to_string(), identity.as_deref().unwrap_or_default());
195                        }
196                        return Ok(response);
197                    }
198                    Err(error) if transport(&error) => last = error,
199                    Err(error) => return Err(error.into()),
200                }
201            }
202            if let Some(db) = &db {
203                let _ = db.endpoint(&ip.to_string(), None, Some("connection failed"), "");
204            }
205            tried_current = false;
206            endpoint = self
207                .live
208                .lock()
209                .unwrap()
210                .backups
211                .pop_front()
212                .map(|(ip, stored)| {
213                    // An explicit --sni applies to every endpoint.
214                    (ip, if self.auto { stored } else { forged.clone() })
215                });
216        }
217        Err(last.into())
218    }

Design points:

Only transport errors trigger it: connect errors and timeouts switch identities and endpoints; HTTP-level errors (such as 403) are returned as they are, never mistaken for censorship.
Explicit choices win: with --sni (auto is false) there is no silent identity switch, and failover keeps your identity on every endpoint; with --ip only that endpoint is used, without failover.
Two levels of memory: the in-process Live struct sends later requests straight to the endpoint and identity that worked; store::set_sni writes the identity to SQLite so the next command remembers it too:

src/store.rsPersisting the identity that worked for an IPL107–114 · 338cd96 ↗

107    /// The fallback identity proved itself: make it the stored one for this IP.
108    pub fn set_sni(&self, ip: &str, sni: &str) -> Result<()> {
109        self.conn.execute(
110            "UPDATE endpoints SET sni=?,available=1,verified_at=? WHERE ip=?",
111            params![sni, now(), ip],
112        )?;
113        Ok(())
114    }

Downloads (the libaria2 path) fall back on both levels as well. aria2 error codes 1/2/6/19 mean TLS/connect failure, timeout, network problem and name-resolution failure—exactly how a reset SNI or a blocked endpoint shows up. On these codes, gdirect first retries under the other identity; if that fails, it marks the endpoint dead and moves to the next verified endpoint, continuing from the data already downloaded (aria2's .aria2 control file makes the resume work naturally). It only gives up when every endpoint has failed, and the next command rescans:

src/download.rsClassifying transport errors and marking endpoints deadL156–172 · 338cd96 ↗

156// TLS handshake, connect and DNS failures: exactly what the fallback route is for.
157fn transport(code: i32) -> bool {
158    matches!(code, 1 | 2 | 6 | 19)
159}
160fn mark_dead(args: &Args) {
161    let (Some(ip), Some(db)) = (args.ip, &args.db) else {
162        return;
163    };
164    if let Ok(db) = crate::store::Store::open(db) {
165        let _ = db.endpoint(
166            &ip.to_string(),
167            None,
168            Some("download transport failure"),
169            "",
170        );
171    }
172}

src/download.rsFailed download: switch identity, then resume on the next endpointL233–289 · 338cd96 ↗

233    let mut sni = args.sni.clone();
234    let mut tried_alt = false;
235    for attempt in 0..3 {
236        let mut code = download(
237            &args,
238            &url,
239            token.as_deref(),
240            sni.as_deref(),
241            progress.as_deref_mut(),
242        )?;
243        loop {
244            if code != 0 && args.auto && !tried_alt && transport(code) {
245                tried_alt = true;
246                sni = match &sni {
247                    Some(_) => None,
248                    None => Some(crate::api::FORGED_HOST.to_owned()),
249                };
250                code = download(
251                    &args,
252                    &url,
253                    token.as_deref(),
254                    sni.as_deref(),
255                    progress.as_deref_mut(),
256                )?;
257            }
258            if code == 0 || !transport(code) || args.backups.is_empty() {
259                break;
260            }
261            // This endpoint is gone: resume the partial file on the next one.
262            mark_dead(&args);
263            let (ip, stored) = args.backups.remove(0);
264            if progress.is_none() {
265                eprintln!(
266                    "Endpoint {} failed; resuming via {ip}",
267                    args.ip.map(|i| i.to_string()).unwrap_or_default()
268                );
269            }
270            args.ip = Some(ip);
271            sni = if args.auto { stored } else { args.sni.clone() };
272            tried_alt = false;
273            code = download(
274                &args,
275                &url,
276                token.as_deref(),
277                sni.as_deref(),
278                progress.as_deref_mut(),
279            )?;
280        }
281        if code != 0 {
282            if transport(code) {
283                mark_dead(&args);
284            }
285            bail!(
286                "download failed: {} (libaria2 code {code}); partial data is kept, rerun to resume",
287                aria2_error(code)
288            );
289        }

Uploads go through deliver too (creating the resumable session and sending every chunk are wrapped in the same fallback), so the API, download and upload paths all have identity fallback and endpoint failover.

4. Custom `libaria2`: Why Patching the C++ Source Was Necessary

For ls and upload, Rust's reqwest with .resolve() is all we need. Downloading multi-gigabyte files, however, requires multi-connection segmented downloads and crash-safe resume, so gdirect embeds aria2 1.37.0 trimmed down to a static HTTP/HTTPS + OpenSSL 3 library (libaria2).

Here is the catch: Google Drive download requests routinely go through cross-domain HTTP 302 redirects (for instance, from www.googleapis.com or drive.google.com to drive.usercontent.google.com and dynamically generated *.googleusercontent.com hosts). In stock aria2, if you only override the initial URL, aria2 falls right back to system DNS as soon as it follows a redirect to a new hostname; and if you rewrite the hostname in the URL to a raw IP, TLS certificate verification fails immediately on the hostname mismatch.

So gdirect applies a ~120-line patch (vendor/aria2-1.37.0.patch plus vendor/gdirect_route.h) when building libaria2, with five changes at the C++ layer:

(1) Allowlist of Google Drive Transfer Domains

vendor/gdirect_route.h gathers every Google Drive hostname under gdirect's control and declares three routing hooks: gdirect_ip is the scanned endpoint; gdirect_sni is the TLS identity (null = real SNI, empty string = no SNI, a hostname = forged identity); and gdirect_hosthdr remembers the real Google hostname under the forged identity, for the HTTP Host header only:

vendor/gdirect_route.hRouting hooks and the Google Drive hostname allowlistL1–17 · 338cd96 ↗

1// gdirect's narrow routing hook for the pinned aria2 1.37.0 build.
2#pragma once
3#include <cstring>
4#include <string>
5extern const char* gdirect_ip;
6// Empty means no override. "" as a set value means "send no SNI"; a hostname
7// means the forged identity: connect and verify as that name while the HTTP
8// Host header carries gdirect_hosthdr, the real Google host.
9extern const char* gdirect_sni;
10extern std::string gdirect_hosthdr;
11inline bool gdirect_host(const std::string& h) {
12  const std::string suffix = ".googleusercontent.com";
13  return h == "drive.google.com" || h == "drive.usercontent.google.com" ||
14         h == "www.googleapis.com" || h == "content.googleapis.com" ||
15         h == "drive.googleapis.com" ||
16         (h.size() > suffix.size() && h.compare(h.size()-suffix.size(), suffix.size(), suffix) == 0);
17}

(2) Intercepting `getaddrinfo` and TLS SNI in `SocketCore.cc`

When SocketCore calls getaddrinfo to open a connection, any hostname matching gdirect_host(host) is replaced with the scanned gdirect_ip (with AI_NUMERICHOST to skip DNS), and so is g.cn under the forged identity—otherwise switching identity would trigger another system DNS lookup. The same hunk takes over tlsSession_->setSNIHostname():

vendor/aria2-1.37.0.patchHooking TLS SNI and getaddrinfo in SocketCore.ccL23–45 · 338cd96 ↗

23@@ -936,7 +942,9 @@
24     // "localhost" will produce TLS alert with GNUTLS.
25     if (tlsctx->getSide() == TLS_CLIENT && !util::isNumericHost(hostname) &&
26         hostname.find(".") != std::string::npos) {
27-      rv = tlsSession_->setSNIHostname(hostname);
28+      const char* sni = gdirect_host(hostname) ? gdirect_sni : nullptr;
29+      rv = (sni && !*sni) ? TLS_ERR_OK :
30+          tlsSession_->setSNIHostname(sni ? std::string(sni) : hostname);
31       if (rv != TLS_ERR_OK) {
32         throw DL_ABORT_EX(fmt(EX_SSL_INIT_FAILURE,
33                               tlsSession_->getLastErrorString().c_str()));
34@@ -1450,6 +1458,12 @@
35   hints.ai_flags = getDefaultAIFlags();
36   hints.ai_flags |= flags;
37   hints.ai_protocol = protocol;
38+  if (host && gdirect_ip && *gdirect_ip &&
39+      (gdirect_host(host) ||
40+       (gdirect_sni && *gdirect_sni && strcmp(host, gdirect_sni) == 0))) {
41+    host = gdirect_ip;
42+    hints.ai_flags |= AI_NUMERICHOST;
43+  }
44   return getaddrinfo(host, service, &hints, resPtr);
45 }

When gdirect_sni is nullptr (the default), the original hostname is sent as SNI;
Under the forged identity (gdirect_sni points to g.cn), the connection name and SNI both become g.cn, and the certificate is verified against g.cn;
With --no-sni (gdirect_sni is the empty string ""), it returns TLS_ERR_OK without calling setSNIHostname, so OpenSSL sends a ClientHello with no SNI extension at all.

(3) Swapping Identities in `Request.cc`: Connect as `g.cn`, Real Host in the `Host` Header

The key step of the forged identity happens in Request::parseUri: when the URL is a controlled Google hostname and the forged identity is on, the URI's host becomes g.cn (so connection, SNI and certificate checks all use g.cn), and the real hostname is saved in gdirect_hosthdr, which HttpRequest prefers when building the Host: header. On the wire you see a harmless g.cn connection, while the Google front end still routes by the real Drive hostname:

vendor/aria2-1.37.0.patchRequest.cc: URI host becomes the forged identity, real host kept for HostL58–86 · 338cd96 ↗

58@@ -150,13 +152,24 @@
59 {
60   currentUri_ = removeFragment(srcUri);
61   uri::UriStruct us;
62-  if (uri::parse(us, currentUri_)) {
63-    us_.swap(us);
64-    return true;
65+  if (uri::parse(us, currentUri_) && us.protocol == "https") {
66+    if (gdirect_sni && *gdirect_sni && gdirect_host(us.host)) {
67+      // Forged identity: the connection, SNI and certificate name become
68+      // gdirect_sni; the real host is kept only for the HTTP Host header.
69+      gdirect_hosthdr = us.host;
70+      us.host = gdirect_sni;
71+      us_.swap(us);
72+      return true;
73+    }
74+    // Real hosts pass through unchanged; a redirect relative to an already
75+    // swapped URI lands on the forged host and keeps its Host header.
76+    if (gdirect_host(us.host) ||
77+        (gdirect_sni && *gdirect_sni && us.host == gdirect_sni)) {
78+      us_.swap(us);
79+      return true;
80+    }
81   }
82-  else {
83-    return false;
84-  }
85+  return false;
86 }

vendor/aria2-1.37.0.patchHttpRequest.cc: the Host header carries the real hostnameL100–107 · 338cd96 ↗

100@@ -190,7 +192,9 @@
101       builtinHds.emplace_back("Accept-Encoding:", acceptableEncodings);
102     }
103   }
104-  builtinHds.emplace_back("Host:", getHostText(getURIHost(), getPort()));
105+  builtinHds.emplace_back("Host:", gdirect_hosthdr.empty()
106+                                       ? getHostText(getURIHost(), getPort())
107+                                       : gdirect_hosthdr);

(4) Stripping `Authorization` Across Redirects & Redirect Allowlisting

When downloading a private file, the initial request goes to https://www.googleapis.com/drive/v3/files/{id}?alt=media with Authorization: Bearer <token>. Once redirected to a pre-signed *.googleusercontent.com URL, forwarding the Authorization header makes Google's content servers reject the request for carrying two sets of credentials—and risks leaking the token across domains. The patch therefore has HttpRequest.cc emit Authorization: only when the real hostname (gdirect_hosthdr when set, otherwise the URI host) is www.googleapis.com. It must check the real hostname rather than the connection name, or the token would be stripped by mistake under the forged identity. Request.cc also requires every redirect target to be https and either match gdirect_host() or be the forged host itself.

(5) Removing the OpenSSL 3 Legacy Provider Dependency

Under OpenSSL 3, stock aria2 unconditionally calls OSSL_PROVIDER_load(nullptr, "legacy") at startup (for RC4 in BitTorrent encryption), which crashes on systems without the legacy provider module installed. Guarding that call with #ifdef ENABLE_BITTORRENT makes the HTTP-only static build self-contained.

5. Certificate Verification Trusts Only Google's Own GTS Roots

Getting past censorship isn't only about establishing a connection; the connection also has to really reach Google. gdirect used to ship a full cacert.pem bundle (hundreds of trusted roots), but a general-purpose bundle means any one of those CAs being compromised or mis-issuing is enough for an attacker to intercept googleapis.com, silently tampering with downloads or capturing OAuth tokens. The public leak of eMudhra intermediate CA keys in October 2026 is a real example—and clients like OpenSSL/webpki don't check revocation, so a leaked key stays usable until it expires.

But Google issues every endpoint we use (googleapis.com, googleusercontent.com, g.cn) under its own Google Trust Services roots. So gdirect embeds those few GTS roots in the binary and trusts only them, rejecting any chain issued by another CA:

src/api.rsDisable built-in CA roots and load only the embedded Google GTS rootsL19–36 · 338cd96 ↗

19/// Google issues every endpoint we touch under its own Google Trust Services
20/// roots. Anchoring verification there (instead of a full CA bundle) means a
21/// compromised or mis-issuing third-party CA cannot impersonate Google for
22/// this client, even with a publicly leaked intermediate key.
23pub(crate) const GTS_PEM: &str = include_str!("gts-roots.pem");
24pub(crate) fn client_builder() -> reqwest::blocking::ClientBuilder {
25    let mut builder = Client::builder().tls_built_in_root_certs(false);
26    const BEGIN: &str = "-----BEGIN CERTIFICATE-----";
27    const END: &str = "-----END CERTIFICATE-----";
28    for block in GTS_PEM.split(BEGIN).skip(1) {
29        let end = block.find(END).expect("bundled GTS root must be complete") + END.len();
30        builder = builder.add_root_certificate(
31            reqwest::Certificate::from_pem(format!("{BEGIN}{}", &block[..end]).as_bytes())
32                .expect("bundled GTS root must parse"),
33        );
34    }
35    builder
36}

On the reqwest side, tls_built_in_root_certs(false) turns off the system bundle and the GTS roots are loaded one by one; on the libaria2 side, the same PEM is written next to the state database and passed to aria2 as ca-certificate:

src/download.rsWriting the embedded GTS roots to disk for libaria2L366–375 · 338cd96 ↗

366fn bundled_ca() -> Option<PathBuf> {
367    let dir = dirs::data_local_dir()?.join("gdirect");
368    fs::create_dir_all(&dir).ok()?;
369    let path = dir.join("gts-roots.pem");
370    match fs::read(&path) {
371        Ok(existing) if existing == crate::api::GTS_PEM.as_bytes() => {}
372        _ => fs::write(&path, crate::api::GTS_PEM).ok()?,
373    }
374    Some(path)
375}

So even if a censor deploys a fake Google certificate issued by a compromised CA, gdirect refuses the handshake. The separate cacert.pem file is gone; --ca-cert remains as an explicit override.

6. Overwrite Guards, Virus-Scan Confirmation Pages, and MD5 Verification

On the Rust side, src/download.rs adds three more safeguards around libaria2.

A safety check for existing files: if the output file already exists without a companion .aria2 control file, gdirect hashes it and compares the result with Drive's md5Checksum—reporting Already downloaded on a match and refusing to overwrite otherwise, so an unrelated file is never treated as a partial download:

src/download.rsNever treat an unrelated existing file as a resumable downloadL207–218 · 338cd96 ↗

207    // Never treat an unrelated existing file as a resumable download.
208    let control = PathBuf::from(format!("{}.aria2", args.output.display()));
209    if args.output.exists() && !control.exists() {
210        ensure!(
211            matches_md5(&args.output, args.md5.as_deref())?,
212            "output already exists; choose another -o path"
213        );
214        if progress.is_none() {
215            println!("Already downloaded {}", args.output.display());
216        }
217        return Ok(());
218    }

Virus-scan confirmation pages for large public files, and verification after saving:

src/download.rsFollowing the HTML confirmation page and verifying MD5L290–323 · 338cd96 ↗

290        let mut prefix = Vec::new();
291        fs::File::open(&args.output)?
292            .take(1024 * 1024)
293            .read_to_end(&mut prefix)?;
294        // Public downloads can return the virus-scan confirmation form with HTTP 200.
295        // Only Drive's own HTML pages are inspected; other files stay untouched.
296        if token.is_none() && html_page(&prefix) {
297            let text = String::from_utf8_lossy(&prefix);
298            if text.contains("download-form") {
299                let next = drive::confirmation(&text, &url)?;
300                ensure!(attempt < 2, "Drive confirmation loop");
301                fs::remove_file(&args.output)?;
302                url = next;
303                continue;
304            }
305            if text.contains("accounts.google.com/ServiceLogin")
306                || text.contains("Google Drive - Quota exceeded")
307                || text.contains("Google Drive - Access denied")
308            {
309                bail!(
310                    "Drive returned a login/quota error page; saved for inspection at {}",
311                    args.output.display()
312                );
313            }
314        }
315        ensure!(
316            args.md5.is_none() || matches_md5(&args.output, args.md5.as_deref())?,
317            "MD5 does not match Drive's checksum; the file is kept at {} for inspection",
318            args.output.display()
319        );
320        if progress.is_none() {
321            println!("Saved {}", args.output.display());
322        }
323        return Ok(());

Only real HTML counts as a confirmation page: when downloading a large shared file without signing in, Google Drive returns an HTTP 200 HTML page containing <form id="download-form">. Only when the file actually starts with <!doctype html or <html does gdirect parse that form, build the real download URL with confirm / uuid, and download again—so archives or source files that happen to contain the string download-form are never touched.
MD5 after saving: once the download completes, the whole file is streamed through MD5 and checked against Drive's metadata, guaranteeing that multi-connection chunk reassembly is exact.

7. 32 MiB Resumable Chunked Uploads

The upload module (src/upload.rs) implements the Google Drive v3 Resumable Upload protocol. Direct cross-border links have high RTT, and tiny chunks waste most of their time waiting for per-chunk acknowledgments, so the chunk size is 32 MiB. Every request goes through deliver from section 3, so uploads get identity fallback and endpoint failover too.

When an upload starts, gdirect looks up any earlier session in local SQLite keyed by (canonical local path, target parent folder ID, remote filename) and asks Google how far it got:

src/upload.rsProbing an existing upload session's committed offset through deliverL85–105 · 338cd96 ↗

85        let mut session = saved.filter(|s| s.size == size && s.modified == modified);
86        if let Some(s) = &session {
87            let url = session_url(&s.url)?;
88            let response = self.deliver(|client, forged| {
89                client
90                    .put(crate::api::gate(forged, &url))
91                    .bearer_auth(&token)
92                    .header("Content-Range", format!("bytes */{size}"))
93                    .header("Content-Length", 0)
94            })?;
95            match response.status() {
96                StatusCode::OK | StatusCode::CREATED => {
97                    let result = response.json()?;
98                    db.unset(&key)?;
99                    return Ok(result);
100                }
101                StatusCode::PERMANENT_REDIRECT => start = offset(&response, size)?,
102                StatusCode::NOT_FOUND | StatusCode::GONE => session = None,
103                _ => return Err(failure(response)),
104            }
105        }

As long as the local file's size and nanosecond modification time haven't changed, re-running the same command after an interruption sends Content-Range: bytes */{size}, reads the committed offset from the Range header of the 308 Permanent Redirect reply, and resumes from there.

8. Localhost Loopback + PKCE OAuth Flow

In src/auth.rs, gdirect login implements the OAuth 2.0 Loopback + PKCE (RFC 7636 S256) flow without any external OAuth SDK:

src/auth.rsEphemeral localhost listener and PKCE authorization URLL196–215 · 338cd96 ↗

196    let listener = TcpListener::bind(("127.0.0.1", 0))?;
197    listener.set_nonblocking(true)?;
198    let redirect = format!(
199        "http://127.0.0.1:{}/callback",
200        listener.local_addr()?.port()
201    );
202    let state = random_string()?;
203    let verifier = random_string()?;
204    let mut url = url::Url::parse("https://accounts.google.com/o/oauth2/v2/auth")?;
205    url.query_pairs_mut().extend_pairs([
206        ("client_id", client_id),
207        ("redirect_uri", redirect.as_str()),
208        ("response_type", "code"),
209        ("scope", SCOPE),
210        ("access_type", "offline"),
211        ("prompt", "consent select_account"),
212        ("state", state.as_str()),
213        ("code_challenge", challenge(&verifier).as_str()),
214        ("code_challenge_method", "S256"),
215    ]);

It binds a temporary non-blocking TCP listener on 127.0.0.1:0, checks the state parameter on the browser callback to prevent CSRF, and exchanges the authorization code plus code_verifier at https://oauth2.googleapis.com/token for a refresh_token. The SQLite database and any rclone credentials exported for mount are created with 0600 permissions, and the exported credentials are deleted on unmount or when a mount fails.

Putting Memoh Under the Knife: Specializing an Agent for Cheaper Telegram Group Chats

Mon, 03 Aug 2026 00:00:00 +0000

A while ago, I deployed Memoh as a few Telegram group-chat bots, mainly to burn through an Ollama subscription I couldn't possibly use up on my own (some idiot subscribed to a full year of Ollama at once).

After running it for a bit, I noticed that ten minutes of group chat could burn through half of my 5-hour limit, and a single day would eat half of my weekly quota. That got me thinking: in a pure chat scenario, how can I drive down both the number of LLM calls and the token consumption per call? I made a set of targeted changes for group chat, and my total cost dropped to less than a fifth of what it used to be.

To be clear, this post isn't here to nitpick upstream. Memoh is a thoughtfully designed general-purpose agent framework—multi-step tool loops, a rich toolset, and structured message metadata are all reasonable and even necessary for an agent that "gets work done for you." What I did was specialize it into a chatbot: an agent in a group chat spends most of its life just talking rather than executing tasks, and "talking" inside a general-purpose agent loop has a very unfavorable cost structure.

Where Does the Money Actually Go?

First, here is the breakdown of a real request captured before my optimizations:

Component	Size	Share
Messages (conversation history)	~157.7 KB	68.0%
Tools (tool schemas)	~49.5 KB	21.3%
System (system prompt)	~24.7 KB	10.7%

A single request is over 200 KB—roughly 50,000 tokens. The catch is what a general-purpose agent's tool loop does with that payload: every extra tool call re-reads all 50,000 tokens from scratch. And chat scenarios are packed with tool calls: sending a message is one, replying is one, sending a sticker is another, and after sending, a delivery status comes back for the model to read and confirm. Behind a single outgoing group-chat message, the full context is read 2 to 3 times over.

So my changes followed three directions: make fewer calls, read less per call, and hit the prompt cache on almost everything already read. Every change below includes the actual code from my fork (mizorewww/Memoh, compared against upstream baseline d0c5c735^).

Change 1: Clock Out Right After Sending

Upstream's loop works like this: the model decides to send a message → calls send → the framework returns the delivery status (message ID, platform, target conversation, etc.) as a tool result → the model is invoked a second time, reads the entire context again, says "okay, sent," and finishes. For a group-chat bot, that final call is almost pure waste: the message has already been delivered to the chat, and there is nothing meaningful left for the model to do except echo a confirmation. With the Doubao model, an even worse failure mode showed up: seeing the success result, the model kept calling send over and over in an infinite loop.

My fix wraps the provider in a decorator (internal/agent/runtime/native/terminal_send.go). Once send successfully delivers to the current conversation, the next time the model would be called, the decorator immediately returns FinishReasonStop locally without making a network request at all:

internal/agent/runtime/native/terminal_send.goEnd the turn locally after successful deliveryL76–103 · ad8012c ↗

76type terminalSendProvider struct {
77	sdk.Provider
78	state *terminalSendState
79}
80
81func (p *terminalSendProvider) DoGenerate(ctx context.Context, params sdk.GenerateParams) (*sdk.GenerateResult, error) {
82	if p.state.isDelivered() {
83		return &sdk.GenerateResult{
84			FinishReason: sdk.FinishReasonStop,
85			Response:     sdk.ResponseMetadata{ID: terminalSendResponseID},
86		}, nil
87	}
88	return p.Provider.DoGenerate(ctx, params)
89}
90
91func (p *terminalSendProvider) DoStream(ctx context.Context, params sdk.GenerateParams) (*sdk.StreamResult, error) {
92	if !p.state.isDelivered() {
93		return p.Provider.DoStream(ctx, params)
94	}
95	parts := make(chan sdk.StreamPart, 2)
96	parts <- &sdk.FinishStepPart{
97		FinishReason: sdk.FinishReasonStop,
98		Response:     sdk.ResponseMetadata{ID: terminalSendResponseID},
99	}
100	parts <- &sdk.FinishPart{FinishReason: sdk.FinishReasonStop}
101	close(parts)
102	return &sdk.StreamResult{Stream: parts}, nil
103}

Of course, the protocol requires every tool_call to be paired with a tool_result, so we can't literally drop the result without leaving a dangling tool call in history. Instead, the result is compressed to the bare minimum: a successful send to the current conversation returns just {"ok": true}, only attaching target metadata when sending across conversations; failures return a structured retryable error (error_code + retryable + guidance) so the model has a chance to recover instead of crashing the loop with a Go error:

internal/agent/tool/message.goMinimal send resultL311–324 · ad8012c ↗

311func (p *MessageProvider) execSend(ctx context.Context, session SessionContext, toolCallID string, args map[string]any) (any, error) {
312	if session.SessionType == sessionmode.Discuss {
313		sendResult, err := p.exec.SendDirectForTool(ctx, toMessagingSession(session), "", toolCallID, args)
314		if err != nil {
315			return messageSendErrorResult(err)
316		}
317		resp := map[string]any{"ok": true}
318		if !session.IsSameConversation(sendResult.Platform, sendResult.Target) {
319			resp["platform"] = sendResult.Platform
320			resp["target"] = sendResult.Target
321			resp["delivered"] = "target"
322		}
323		return resp, nil
324	}

There is one more layer in history projection. When future turns read the conversation history, successful send calls are folded into a minimal call/result pair containing only the five audience-visible fields (text, reply_to, sticker_id, attachments, message), stripping delivery internals (bot_id, platform, message_id, etc.) from the prompt context. Note that this folding happens only in the projection layer—the persisted canonical history remains untouched for debugging:

internal/agent/application/service_messages.goMinimal send call in history projectionL232–246 · ad8012c ↗

232func minimalDiscussSendCall(call sdk.ToolCallPart) sdk.ToolCallPart {
233	input := normalizeToolInputMap(call.Input)
234	minimal := make(map[string]any, 5)
235	// Current-conversation routing is implicit. Keep only audience-visible
236	// payload fields and the reply reference needed to understand the exchange.
237	for _, key := range []string{"text", "reply_to", "sticker_id", "attachments", "message"} {
238		if value, ok := input[key]; ok {
239			minimal[key] = value
240		}
241	}
242	return sdk.ToolCallPart{
243		ToolCallID: strings.TrimSpace(call.ToolCallID),
244		ToolName:   "send",
245		Input:      minimal,
246	}

The result: on pure messaging turns, LLM calls drop from 2 to 1—an immediate 50% savings. Multi-step workflows that search the web or call other tools still work normally; the bot simply treats send as the terminal action of the turn.

Change 2: Send Text, Reply, and Sticker in a Single Call

In group chats, a single response is often both a reply to someone and accompanied by a sticker. Previously, that required multiple tool calls, re-reading the entire context each time. I merged them into a single send tool signature:

send(text[optional], reply_to[optional], sticker_id[optional])

Concretely, this involved two changes. First, upstream already had a top-level reply_to parameter alongside a redundant nested message.reply{message_id} object; I removed the nested form, kept only the top-level parameter, and restricted it to message IDs visible in the current turn (returning a retryable error if invisible instead of replying to the wrong message).

Second, I had previously written an MCP server for sending stickers, only to realize that invoking an external MCP tool added an extra LLM round-trip every time. I folded the sticker catalog directly into the sticker_id parameter of send:

internal/agent/runtime/native/telegram_sticker.goTurn the sticker MCP into an internal backend of sendL14–31 · ad8012c ↗

14	"github.com/memohai/memoh/internal/delivery"
15)
16
17const (
18	telegramStickerSendGuidance     = " For Telegram, normally pair every coherent public message with at least one context-appropriate Sticker by setting `sticker_id` in the same send call; omit it only when the catalog has no suitable Sticker. Choose from the full visual descriptions, not from the original emoji alone, and avoid repetitive duplicates."
19	telegramStickerCatalogSchemaKey = "x-memoh-sticker-catalog"
20)
21
22type telegramStickerSchemaEntry struct {
23	ID          string `json:"id"`
24	Description string `json:"description"`
25	Emoji       string `json:"emoji"`
26	Status      string `json:"status"`
27}
28
29// mergeTelegramStickerSendTools converts the Sticker MCP into an internal
30// backend for the first-party send tool. The model sees one send call whose
31// sticker_id enum contains the complete, stable catalog; it never sees a

Within a single send call, the bot delivers the text first and then the sticker; if the sticker fails after the text is already sent, it returns partial success and terminates immediately so the model won't retry and send duplicate text.

I also had a vision model pre-generate a concise visual description (under 50 characters) for every sticker, stored alongside the sticker-set metadata in SQLite. Sorted deterministically by ID, these descriptions are embedded into the send tool schema's description. Once merged, sending a sticker incurs virtually zero extra LLM cost.

Change 3: Move Everything That Changes to the Tail of the Prefix

This is the key to prompt-cache hit rates. Every provider's prompt caching mechanism (Anthropic, OpenAI, DeepSeek, Kimi—all of them) works by prefix matching: the prefix of a request must match byte-for-byte, and the first byte that changes invalidates the cache for everything after it.

To be fair, upstream's default behavior isn't bad across ordinary turns: conversation history is appended at the end, so automatic prefix caches like DeepSeek's catch most of it. The real leak was a frequently rewritten file buried in the middle of the system prompt: upstream placed MEMORY.md between AGENTS.md and PROFILES.md. Because MEMORY.md is rewritten constantly by the memory subsystem, every update invalidated the cache for everything following it. My fix was a one-function change:

internal/agent/runtime/native/fs.goMove MEMORY.md to the tail of the system promptL66–71 · ad8012c ↗

66// systemFileLoadOrder keeps the frequently changing memory summary at the
67// tail of the system prompt. Stable instruction/profile content therefore
68// remains in the reusable prefix when MEMORY.md changes between turns.
69func systemFileLoadOrder() []string {
70	return []string{"AGENTS.md", "PROFILES.md", "MEMORY.md"}
71}

Changing the load order from AGENTS.md → MEMORY.md → PROFILES.md to AGENTS.md → PROFILES.md → MEMORY.md keeps the most stable instructions at the front of the prefix and pushes the most volatile file to the very end of the system prompt. Following the same principle, I cleaned up several other sources of "prefix pollution":

The memory-writing manual (~2.6 KB) is now injected only when a memory provider is actually configured; when set to None, the entire section is omitted:

internal/agent/application/service.goSkip memory instructions when no memory provider is configuredL710 · ad8012c ↗

710MemoryInstructions:    strings.TrimSpace(botSettings.MemoryProviderID) != "",

Platform identity now injects only the active platform's fields, and strips avatar_url (which contained the bot token) from the injected prompt.

The difference is measurable: under the old "dynamically mutate system prompt" path, a forced-reply call measured just 5.1% cache read (a single measurement of that specific code path, not the everyday baseline). After the fix, two consecutive requests in a live group chat sent 52,259 and 52,633 input tokens, of which 52,096 and 52,480 were served from cache—a 99.7% cache hit rate, leaving only ~160 uncached input tokens billed at full price per turn.

Change 4: Invisible Request Slimming

Smaller requests mean cheaper reads on every turn. This part stacks several small reductions together, all exposed as configurable Telegram channel policies stored in bot metadata:

internal/agent/channelpolicy/policy.goFour Telegram channel policy keysL21–24 · ad8012c ↗

21TelegramToolCallsEnabledMetadataKey = "telegram_tool_calls_enabled"
22TelegramEnabledToolsMetadataKey     = "telegram_enabled_tools"
23TelegramSkillsEnabledMetadataKey    = "telegram_skills_enabled"
24TelegramMessageMetadataModeKey      = "telegram_message_metadata_mode"

telegram_tool_calls_enabled is the master switch for tools, telegram_enabled_tools is an explicit allowlist (an empty array disables all tools), and telegram_skills_enabled controls skills. A chat bot doesn't need shell execution or file editing; on my two production bots I enabled only send, turning off 37 tools. The crucial detail is when filtering happens: the allowlist runs before schema generation, so disabled tools don't contribute a single byte to the request:

internal/agent/runtime/native/agent.goFilter tools before schema generationL990–997 · ad8012c ↗

990}
991filtered := make([]sdk.Tool, 0, len(providerTools))
992for _, tool := range providerTools {
993	if cfg.ChannelPolicy.AllowsTool(tool.Name) || cfg.ChannelPolicy.AllowsBackendTool(tool.Name) {
994		filtered = append(filtered, tool)
995	}
996}
997providerTools = filtered

Before optimization, the Tools section accounted for 21.3% of the request (~49.5 KB, roughly 12,000 tokens); now it is almost completely gone. Policy evaluation is fail-closed: if reading the policy fails, it exposes no tools rather than leaking tools that should be disabled.

telegram_message_metadata_mode controls how message metadata headers are projected, defaulting to compact on Telegram. Upstream prepends a full metadata header to every message sent to the model: <message id="..." sender="..." t="..." channel="telegram" conversation="..." type="..." target="...">. The canonical format in SQLite stays unchanged, but the transient projection sent to the LLM strips the per-message repeated timestamp, platform, and conversation name while keeping speaker attribution for group chats:

internal/agent/turn/user_header.goCompact message header projectionL165–183 · ad8012c ↗

165func ProjectUserMessageHeader(content, mode string) string {
166	if strings.ToLower(strings.TrimSpace(mode)) != "compact" || !strings.HasPrefix(strings.TrimSpace(content), "<message") {
167		return content
168	}
169	start := strings.Index(content, "<message")
170	if start < 0 {
171		return content
172	}
173	endRel := strings.Index(content[start:], ">")
174	if endRel < 0 {
175		return content
176	}
177	end := start + endRel
178	compactOpening, ok := projectMessageOpeningTag(content[start : end+1])
179	if !ok {
180		return content
181	}
182	return content[:start] + compactOpening + content[end+1:]
183}

That saves 85–120 characters per message, reduces occurrences of channel="telegram" in a request from 148 down to 2, and shrinks a full-history payload from 221.0 KB to 208.7 KB.

Change 5: Giving Text-Only Models "Eyes"

Some of the primary models I run don't support vision (such as DeepSeek V4 Flash), yet people post images in group chats all day long. I added an auxiliary vision model mechanism: when the primary model declares no vision support and the current turn includes image attachments, a cheap small vision model describes the images first:

internal/agent/application/auxiliary_vision.goTrigger only when the primary model lacks visionL183–196 · ad8012c ↗

183func (s *Service) describeImagesWithAuxiliaryVision(
184	ctx context.Context,
185	req ChatRequest,
186	primarySupportsVision bool,
187	attachments []gatewayAttachment,
188) string {
189	if s == nil || primarySupportsVision {
190		return ""
191	}
192	images := extractNativeImageParts(attachmentsToAny(attachments))
193	return s.describeImagePartsWithAuxiliaryVision(ctx, req, primarySupportsVision, images)
194}
195
196func (s *Service) describeImagePartsWithAuxiliaryVision(

The generated description is appended as an <auxiliary_vision_description> block at the tail of the last user message, once again leaving the cached prefix untouched:

internal/agent/application/auxiliary_vision.goAppend vision descriptions to the tail of the last user messageL454–467 · ad8012c ↗

454func appendAuxiliaryVisionToLastUserMessage(messages []sdk.Message, visionContext string) []sdk.Message {
455	visionContext = strings.TrimSpace(visionContext)
456	if visionContext == "" {
457		return messages
458	}
459	for i := len(messages) - 1; i >= 0; i-- {
460		if messages[i].Role != sdk.MessageRoleUser {
461			continue
462		}
463		messages[i].Content = append(messages[i].Content, sdk.TextPart{Text: "\n\n" + visionContext})
464		return messages
465	}
466	return append(messages, sdk.UserMessage(visionContext))
467}

Bonus Telegram-Specific Polish

A few additional improvements aimed at making the bot feel more natural in Telegram group chats also helped keep costs predictable:

Continuously refreshing the typing indicator throughout any turn where the bot is selected to reply, so group members see "is typing…" instead of 30 seconds of dead silence followed by a wall of text;
Switching context compaction to a rolling-summary mode whose prompt explicitly forbids repeating raw tool-argument JSON; the context budget takes the smaller of the model's context window and the bot's compaction threshold, and truncation boundaries never split a user/tool pair mid-exchange.

How Much Did It Actually Save?

Modeling the savings is straightforward. The input cost of a single LLM call is:

C_{\mathrm{call}} = T \times \left[(1-h) \times P_{\mathrm{in}} + h \times P_{\mathrm{cache}}\right] = T \times P_{\mathrm{in}} \times \left[(1-h) + h \times r\right]

where T is the number of input tokens, h is the cache hit rate, and r = P_cache / P_in is the ratio of cached-input price to uncached-input price. Multiplying by the number of calls N per turn separates the before/after ratio into three independent factors:

\frac{C_{\mathrm{before}}}{C_{\mathrm{after}}} = \frac{N_{\mathrm{old}}}{N_{\mathrm{new}}} \times \frac{T_{\mathrm{old}}}{T_{\mathrm{new}}} \times \frac{f(h_{\mathrm{old}})}{f(h_{\mathrm{new}})}

where $f(h) = (1-h) + h \times r$ . Each of the three factors is grounded in real measurements:

N_old / N_new = 2–3: pure text turns drop from 2 calls to 1, and reply + sticker turns drop from 3 calls to 1. This is structural—no estimation involved;
T_old / T_new ≈ 70K / 52K ≈ 1.35: stripping the unused tool schemas (49.5 KB), dynamic sticker catalog (18.3 KB), and repeated message headers removes ~20K tokens per request;
h_new = 99.7% is measured in production; I set h_old = 95% to be generous to upstream's automatic prefix caching, which works well on ordinary turns and only loses its prefix when MEMORY.md is rewritten.

Using official provider pricing as of 2026-08-03—DeepSeek V4 Flash at $0.0028/$0.14 = 1/50, Claude Sonnet 4.6 at 1/10, Kimi K2.6 at ~1/6, and Doubao seed-1.6 at 1/5—and plugging in conservative numbers (N ratio = 2, h_old = 95%):

Model	r	f(95%) / f(99.7%)	Total Factor 2 × 1.35 × f-ratio	Savings
DeepSeek V4 Flash	1/50	3.0	≈ 8.1×	≈ 88%
Claude Sonnet 4.6	1/10	1.4	≈ 3.8×	≈ 74%
Kimi K2.6	≈1/6	1.2	≈ 3.3×	≈ 70%

The pattern is obvious: the steeper the cache discount, the more leverage you get from pushing the hit rate to 99.7%—DeepSeek's 50× cache discount pushes total savings close to 90%. Meanwhile, the call-count factor N doesn't depend on caching at all; it's free money. This matches what I see on my actual billing dashboard: costs dropped to well under 1/5 of what they were, and the conservative formula gives ~1/8 on DeepSeek even before counting the eliminated output tokens.

Closing Thoughts

All I really did was ask one question at every step of the pipeline: when you put a general-purpose agent framework into a high-frequency group chat, is this read or this LLM call actually necessary? Don't make an extra call just to confirm a sent message, combine text + reply + sticker into a single call, push volatile files to the tail of the prompt prefix, don't send a single byte of unused tool schemas, and let a cheap small vision model look at images first.

Once again, credit goes to upstream for building a solid foundation: the tool loop, history projection, and channel abstractions are cleanly separated. The reason I could pull off this degree of specialization in a little over a week is precisely because those boundaries were well drawn—wrapping the provider for terminal send without touching the core loop, and filtering tools before schema generation, both plugged cleanly into seams the architecture already had. If your use case is "an agent that does multi-step work," upstream's defaults are probably right for you; if, like me, you just want a talkative resident in your Telegram group that won't burn through your wallet, I hope these notes help.

New Blog Features: Language Icons, Source Quotes, Tickers, and Charts

Mon, 22 Jun 2026 00:00:00 +0000

This post was written during the old blog (Next.js + MDX) era. After migrating to the current static site, all of these features happen at build time: TradingView widgets became ticker cards that only load the live chart when you click "Live chart"; ECharts blocks became static SVG charts with hover values; source embeds are pinned commit snippets with line numbers; and language icons became corner language labels on code blocks.

This update pushes the blog's writing experience another step forward: code blocks are easier to scan, code references link straight to GitHub, and financial posts can embed TradingView charts with a tiny shortcode.

What's New

Language Icons on Code Blocks

A language logo and copy button float in the top-right corner of every code block—no title bar, no badge border, and no extra vertical space. When a block has a title or comes from GitHub, a small path line appears above the code and the GitHub link sits tucked into a small icon in the top-right corner.

const feature = 'language icons'
const theme = 'catppuccin'
console.log(feature, theme)

Restyled Inline Code

Inline code in prose like git log --follow, $AAPL, and ::tv AAPL no longer sticks out like a bright patch. In light mode it blends closer to Catppuccin Latte's paper tone, and in dark mode it settles into a low-luminance Mocha surface.

Quoting GitHub Code Directly

When writing a post, enter:

::github-code repo="mizorewww/blog" ref="c067042276d4b4b384c66c0c61fcd4f6716eb599" path="contentlayer.config.ts" lines="251-253" lang="ts" title="contentlayer.config.ts"

It renders as a code block with Shiki syntax highlighting, a language icon, and a direct link to GitHub:

contentlayer.config.tsL251–253 · c067042 ↗

251const githubEmbedPattern = /^::github-(code|diff)\s+(.+)$/
252const tradingViewMiniPattern = /^\$([A-Za-z0-9._:-]+)$/
253const tradingViewAdvancedPattern = /^::(?:tv|tv-advanced|tradingview)\s+(.+)$/

Line-by-Line Diff Highlighting Like GitHub

When writing a post, enter:

::github-diff repo="mizorewww/blog" ref="c067042276d4b4b384c66c0c61fcd4f6716eb599" path="css/prism.css" lines="1-15"

Rendered output:

Changes to css/prism.css in c067042

@@ -8,6 +8,44 @@ figure[data-rehype-pretty-code-figure] [data-rehype-pretty-code-title] {
   @apply flex items-center justify-between gap-3 border-b border-slate-200 bg-white/70 px-4 py-2 text-xs text-slate-600 dark:border-[#405064] dark:bg-white/[0.035] dark:text-white/65;
 }
 
+figure[data-rehype-pretty-code-figure] .code-title-main {
+  @apply inline-flex min-w-0 items-center gap-2;
+}
+
+figure[data-rehype-pretty-code-figure] .code-language-icon {
+  @apply inline-flex h-5 min-w-5 shrink-0 items-center justify-center rounded-[5px] border border-slate-300 bg-slate-100 px-1 font-sans text-[0.62rem] leading-none font-semibold text-slate-600 dark:border-white/15 dark:bg-white/10 dark:text-white/70;
+}
+
+figure[data-rehype-pretty-code-figure] .code-language-icon[data-code-language='ts'],
+figure[data-rehype-pretty-code-figure] .code-language-icon[data-code-language='tsx'] {
+  @apply border-sky-200 bg-sky-50 text-sky-700 dark:border-sky-400/25 dark:bg-sky-400/10 dark:text-sky-200;

diff

Single-Line Tickers Become Mini Charts

Put a ticker on its own line:

$AAPL
$BINANCE:BTCUSDT.P

On the old site, each of these lines turned into a Mini Chart. The current static site doesn't load TradingView scripts upfront—instead, it renders a ticker card linking to TradingView:

NASDAQ:AAPLApple Inc.TradingView ↗

BINANCE:BTCUSDT.PBitcoin / USDT PerpetualTradingView ↗

Shortcode for Advanced Charts

When writing a post, enter:

::tv AAPL interval=60 height=460

On the old site this rendered an interactive Advanced Chart; here it is likewise a ticker card.

Rendering ECharts Directly from Code Blocks

Use an ```echarts code block whose body is the chart's option JSON, with title and height supported in the meta string:

```echarts title="Weekly Visits" height=320
{
  "tooltip": { "trigger": "axis" },
  "xAxis": {
    "type": "category",
    "data": ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
  },
  "yAxis": { "type": "value" },
  "series": [
    {
      "name": "Visits",
      "type": "line",
      "smooth": true,
      "areaStyle": {},
      "data": [120, 200, 150, 80, 70, 110, 130]
    }
  ]
}
```

On the old site these were two ECharts instances. The current site draws the same data as static SVG at build time; since there is no pie chart type, the second example is represented as a data table:

Weekly Visits

050100150200TueThuSat200 · Tue

Posts by Category

Category	Posts	Share
Tinkering	40	40%
Dev	32	32%
Notes	18	18%
Other	10	10%

If the JSON is invalid, it gracefully falls back to a regular code block instead of breaking the build. When you need functions or other options that plain JSON cannot express, you can also pass a JS object directly to the <ECharts> component. Charts follow the site's light/dark theme, and the echarts library is lazy-loaded on demand so it stays out of the initial bundle.

How It Works Under the Hood

Turning Shortcodes into Components in Remark

These regular expressions match the ticker chart syntax. $AAPL becomes a Mini Chart and ::tv AAPL becomes an Advanced Chart.

contentlayer.config.tsL251–253 · c067042 ↗

251const githubEmbedPattern = /^::github-(code|diff)\s+(.+)$/
252const tradingViewMiniPattern = /^\$([A-Za-z0-9._:-]+)$/
253const tradingViewAdvancedPattern = /^::(?:tv|tv-advanced|tradingview)\s+(.+)$/

The Mini Chart transform is intentionally conservative: it only triggers when an entire paragraph consists of a single ticker, so mentioning $AAPL in the middle of a sentence won't drop a giant chart into the paragraph.

contentlayer.config.tsL288–295 · c067042 ↗

288function createTradingViewMiniNode(symbol: string): MdastNode | null {
289  if (!isTradingViewTicker(symbol)) {
290    return null
291  }
292
293  return createMdxFlowNode('TradingViewMiniChart', {
294    symbol: normalizeTradingViewSymbol(symbol),
295  })

Advanced Chart adds parameter parsing so attributes like height=460 and interval=60 are forwarded as JSX props.

contentlayer.config.tsL298–313 · c067042 ↗

298function createTradingViewAdvancedNode(source: string): MdastNode | null {
299  const [rawSymbol, ...attrParts] = source.trim().split(/\s+/)
300
301  if (!rawSymbol || !isTradingViewTicker(rawSymbol)) {
302    return null
303  }
304
305  const attrs = parseEmbedAttributes(attrParts.join(' ')) as TradingViewAttrs
306
307  return createMdxFlowNode('TradingViewAdvancedChart', {
308    height: attrs.height,
309    interval: attrs.interval,
310    locale: attrs.locale,
311    symbol: normalizeTradingViewSymbol(rawSymbol),
312    timezone: attrs.timezone,
313  })

Finally, the plugins are wired into Contentlayer's remark pipeline. Because shortcodes are transformed before MDX compilation, no string scanning happens in the browser.

contentlayer.config.tsL1063–1069 · c067042 ↗

1063export default makeSource({
1064  contentDirPath: 'data',
1065  documentTypes: [Blog, Authors],
1066  mdx: {
1067    cwd: process.cwd(),
1068    remarkPlugins: [remarkGfm, remarkTradingViewWidgets, remarkIconShortcodes, remarkGitHubEmbeds],
1069    rehypePlugins: [

Loading TradingView Only on the Client

The official TradingView widget requires injecting an external <script>, so the component stays a client component. On render, it clears the container and inserts the widget mount node and config script; whenever the theme or props change, it rebuilds the widget.

components/TradingViewWidgets.tsxL83–101 · c067042 ↗

83useEffect(() => {
84  const container = containerRef.current
85
86  if (!container) {
87    return
88  }
89
90  container.innerHTML = '<div class="tradingview-widget-container__widget"></div>'
91
92  const script = document.createElement('script')
93  script.async = true
94  script.src = scriptSrc
95  script.text = JSON.stringify(config)
96  container.appendChild(script)
97
98  return () => {
99    container.innerHTML = ''
100  }
101}, [config, scriptSrc])

Ticker symbols are normalized first. Bare AAPL defaults to NASDAQ:AAPL, while an explicit symbol like NYSE:IBM is left untouched.

lib/tradingview.tsL4–20 · c067042 ↗

4export function normalizeTradingViewSymbol(value: string, defaultExchange = 'NASDAQ') {
5  const symbol = value.trim().replace(/^\$/, '').toUpperCase()
6
7  if (!symbol) {
8    return ''
9  }
10
11  if (explicitTradingViewSymbolPattern.test(symbol)) {
12    return symbol
13  }
14
15  if (plainTickerPattern.test(symbol)) {
16    return `${defaultExchange}:${symbol}`
17  }
18
19  return symbol
20}

Enhancing Code Block Title Bars in Rehype

Language icons aren't written by hand in each post; they're injected across the board during the rehype pass. It locates the figure generated by Shiki and ensures a title bar exists.

contentlayer.config.tsL850–868 · c067042 ↗

850function enhanceCodeTitle(node: HastNode) {
851  if (
852    node.tagName !== 'figure' ||
853    node.properties?.['data-rehype-pretty-code-figure'] === undefined
854  ) {
855    return
856  }
857
858  const title = ensureCodeTitle(node)
859  const code = findDescendantElement(node, 'code')
860  const sourceUrl = typeof code?.data?.githubSourceUrl === 'string' ? code.data.githubSourceUrl : ''
861  const language = getCodeLanguage(code)
862  const titleText = getNodeText(title) || getLanguageName(language)
863
864  title.children = [createCodeTitleNode(titleText, language)]
865
866  if (sourceUrl) {
867    title.children.push(createCodeSourceLinkNode(sourceUrl, language))
868  }

The left side of the title bar pairs the language icon with the title text. When no explicit title is provided, it falls back to the language display name—such as TypeScript, Diff, or Markdown.

contentlayer.config.tsL787–799 · c067042 ↗

787function createCodeTitleNode(titleText: string, language: string): HastNode {
788  return {
789    type: 'element',
790    tagName: 'span',
791    properties: { className: ['code-title-main'] },
792    children: [
793      createCodeLanguageIconNode(language),
794      {
795        type: 'element',
796        tagName: 'span',
797        properties: { className: ['code-title-text'] },
798        children: [{ type: 'text', value: titleText || getLanguageName(language) }],
799      },

Visually, the logos come straight from simple-icons brand icons, tree-shaken on demand so they don't bloat first-load JavaScript; languages without a brand icon fall back to a short text badge.

css/prism.cssL11–17 · c067042 ↗

11figure[data-rehype-pretty-code-figure] .code-title-main {
12  @apply inline-flex min-w-0 items-center gap-2;
13}
14
15figure[data-rehype-pretty-code-figure] .code-language-icon {
16  @apply inline-flex h-5 min-w-5 shrink-0 items-center justify-center rounded-[5px] border border-slate-300 bg-slate-100 px-1 font-sans text-[0.62rem] leading-none font-semibold text-slate-600 dark:border-white/15 dark:bg-white/10 dark:text-white/70;
17}

Keeping Inline Code from Stealing Attention

Inline code now uses a low-contrast, subtly bordered token style. It's still unmistakably code, without popping out of the paragraph like a button.

css/tailwind.cssL110–124 · c067042 ↗

110& :where(code):not(pre code) {
111  border: 1px solid color-mix(in oklab, var(--color-slate-300) 72%, transparent);
112  border-radius: 6px;
113  background: linear-gradient(
114    to bottom,
115    color-mix(in oklab, white 92%, var(--color-slate-100)),
116    color-mix(in oklab, var(--color-slate-100) 88%, white)
117  );
118  box-shadow: inset 0 -1px 0 color-mix(in oklab, var(--color-slate-300) 45%, transparent);
119  color: #4c4f69;
120  padding: 0.08rem 0.34rem;
121  font-size: 0.86em;
122  font-weight: 500;
123  line-height: 1.75;
124  word-break: break-word;

GitHub Code and Diffs Still Go Through Shiki

At build time, ::github-code and ::github-diff fetch the real source file or diff and emit a standard code AST node. That means they don't need a custom renderer—Shiki highlights them just like any other code block.

contentlayer.config.tsL543–552 · c067042 ↗

543if (kind === 'code') {
544  const value = selectCodeLines(await getGitHubFile(attrs), attrs.lines)
545  const title =
546    attrs.title ||
547    `${normalizeGitHubRepo(attrs.repo)}:${attrs.path}${attrs.lines ? `#L${attrs.lines}` : ''}`
548
549  return createCodeNode(value, attrs.lang || 'text', title, {
550    showLineNumbers: true,
551    sourceUrl: getGitHubCodeUrl(attrs),
552  })

Diffs follow the exact same path, except the data source is a patch and the language defaults to diff.

contentlayer.config.tsL555–563 · c067042 ↗

555const value = selectCodeLines(await getGitHubDiff(attrs), attrs.lines)
556const title =
557  attrs.title ||
558  `${normalizeGitHubRepo(attrs.repo)}:${attrs.path || `${attrs.base || attrs.ref}...${attrs.head || ''}`}`
559
560return createCodeNode(value || 'No diff matched this query.', attrs.lang || 'diff', title, {
561  showLineNumbers: true,
562  sourceUrl: getGitHubDiffUrl(attrs),
563})

That's why GitHub embeds get the Shiki color theme, language icon, copy button, GitHub link, and diff line backgrounds all at once.

The Mental Model for Writing Posts

For normal posts, you just write Markdown. When you want an icon, write :icon-code:; for a market ticker, write $AAPL or ::tv AAPL; for a data chart, write an ```echarts block; to quote source code, write ::github-code; and to quote a commit diff, write ::github-diff.

The build step turns these shortcodes into stable MDX components or Shiki code blocks, while the runtime only handles interactivity and external widgets. The authoring syntax stays brief, and the rendered page stays predictable.

Fixing Obsidian Web Clipper Not Working on KDE Plasma Wayland

Sun, 10 May 2026 00:00:00 +0000

While using Obsidian on Plasma Wayland, I noticed that whenever Obsidian was already open, clicking Add to Obsidian in Chrome's Obsidian Web Clipper extension created a completely blank note.

The content would only show up properly if I closed Obsidian completely before clipping from Web Clipper.

Cause

This isn't because Web Clipper failed to capture the page—it's because KDE prevented the already-running Obsidian instance from properly receiving focus.

When Web Clipper hands page content over to Obsidian, Obsidian needs to be raised and focused properly to read the incoming data. Under KDE's default window management rules, an Obsidian instance running in the background can be blocked from stealing focus, so the clipped content is never written and you end up with an empty note.

Analysis

The key symptoms are:

Clipping works normally when Obsidian is closed;
Clipping produces an empty note when Obsidian is already open;
This means the browser extension itself is working fine;
The failure happens when Obsidian is already running and KDE prevents it from responding properly to Web Clipper's activation request.

So this is a window focus and activation permission issue in KDE, not a misconfiguration in your Obsidian vault, templates, or Chrome extension.

Solution

Add a window rule for Obsidian in KDE: open System Settings → Window Management → Window Rules and configure it as shown below.

Xiaomi Book Pro 14 on Linux

Sun, 10 May 2026 00:00:00 +0000

The reason I picked up this Xiaomi Book Pro 14 is quite simple: I already have a MacBook Pro that works great for my daily needs, but there are always things that a Mac either can't do or does awkwardly. I needed an x86 thin-and-light to handle the tasks that are "best done on x86."

As someone who can't stand leaving things stock, the first order of business with a new machine was, naturally, installing Arch Linux. I ran into a few pitfalls along the way and thoroughly dug into the fingerprint feature—something I’d never used before. I’m documenting it all here as a record for myself and a reference for anyone who encounters similar issues.

First, here are the specs of my unit:

SoC: Intel Ultra X7 358H
RAM: 32GB LPDDR5x 9600MT/s (Soldered)
GPU: Intel Arc B390

Problem 1: Screen Artifacts

Right after the installation, the screen was glitching out. After some digging, the culprit turned out to be Intel's Panel Self Refresh technology.

The fix is straightforward—just add this line to your kernel parameters:

xe.enable_psr=0

Problem 2: Keyboard Not Recognized

This one was even weirder: the keyboard worked perfectly in the BIOS and GRUB, but became completely unresponsive as soon as the system booted. Getting stuck here during the installation process is incredibly frustrating.

The solution, again, is to add kernel parameters:

i8042.nopnp=1 i8042.dumbkbd=1

Once the screen and keyboard were sorted out, the system was basically usable. What follows is the part I spent the most time on—the fingerprint reader.

Fingerprint: The Main Event

Honestly, I'd never used a device with a fingerprint reader before, so I explored this area quite meticulously and hit many snags. It deserves its own write-up.

Why doesn't `sudo` pop up a fingerprint GUI like macOS?

This was the most irritating part of the whole ordeal. On macOS, sudo can use Touch ID. Why not on Linux?

It's not that "no one thought of it"; it's that the architectures are fundamentally different.

The authentication chain for sudo looks like this:

sudo → PAM → Terminal session or askpass

The beautiful KDE fingerprint authentication dialog, on the other hand, follows a different path:

App → Polkit → KDE Polkit Agent → PAM → fprintd

That GUI is provided by the Polkit agent. It doesn't spring forth from PAM itself, nor is it part of sudo. The PAM module can only participate in the authentication flow; it can't magically transform a terminal-based sudo prompt into a KDE-style identity verification pop-up.

Using sudo -A with SUDO_ASKPASS can indeed pop up a graphical password box, but that's still on sudo's own askpass track, which is separate from the KDE Polkit agent. So, at best, it makes the "enter password" step a GUI, but it can't bring over the fingerprint experience.

macOS can do it because Apple integrated sudo, PAM, Touch ID, the Secure Enclave, and LocalAuthentication at the system level right from the start. On Linux, these components belong to different projects and thus have no natural path to a unified experience.

A Key Limitation of `pam_fprintd`: The PAM Stack is Serial

There's another fundamental limitation with pam_fprintd: in a standard PAM stack, it and password authentication execute in series.

My initial ideal experience was something like this:

For a single sudo prompt, it would simultaneously wait for my password and my fingerprint, using whichever was provided first.

But in reality, this isn't possible. A login manager can use more complex methods to fire up multiple PAM sessions to simulate concurrency, but a regular sudo won't do that.

So, the real-world choices are limited to the following experiences:

Option A: Fingerprint first, fallback to password on failure.
Option B: Password first, fallback to fingerprint if you press Enter with an empty field.
Option C: Only use the Polkit GUI, don't force fingerprint on sudo.
Option D: Use a third-party hack module (not recommended—don't mess around with root authentication).

I eventually settled on Option B, but I tried Option A first. Let me explain why it flopped.

Option A: Fingerprint First, Fallback to Password (Post-Mortem)

My initial, intuitive configuration was this, in /etc/pam.d/sudo:

#%PAM-1.0

auth        sufficient  pam_fprintd.so max-tries=3 timeout=8
auth        include     system-auth
account     include     system-auth
session     include     system-auth

Logically, this looks fine: try the fingerprint first. If it succeeds, great; if not, it'll ask for a password.

But in practice, it broke. I deliberately let the fingerprint recognition fail, then typed the correct password, but sudo still would not let me through. The logs showed entries like this:

pam_unix(sudo:auth): conversation failed
pam_unix(sudo:auth): auth could not identify password for [...]
pam_faillock(sudo:auth): Consecutive login failures ... account temporarily locked

Meanwhile, fprintd was also complaining:

Device was already claimed
Verification was in progress, stopping it

It seems the interaction between sudo, the terminal PAM conversation, pam_fprintd, and the subsequent password stack gets into a messy state. After the fingerprint failed, the password fallback didn't take over cleanly, and it also increased the faillock failure count, temporarily locking my account.

The biggest danger with this kind of problem is that you're modifying the root privilege escalation entry point. If you bork /etc/pam.d/sudo and don't have another root channel, it's easy to lock yourself out.

My recovery method at that time was using a still-functional Polkit:

pkexec sh -c 'cat > /etc/pam.d/sudo <<EOF
#%PAM-1.0
auth        include     system-auth
account     include     system-auth
session     include     system-auth
EOF
faillock --user username --reset
systemctl restart fprintd
'

bash

I suggest keeping this rollback method in mind: first, revert sudo to Arch's default password-only path, then clear faillock, and finally restart fprintd to release any lingering device claims. It's a lifesaver in critical moments.

Option B: Password First, Empty Enter for Fingerprint (My Current Setup)

I later switched to a different approach: the system presents a normal sudo password prompt, but if you just press Enter (submitting an empty password), it falls back to fingerprint verification.

My current configuration in /etc/pam.d/sudo:

#%PAM-1.0

# Prompt for password first; an empty password triggers a fingerprint attempt
auth        sufficient  pam_unix.so try_first_pass likeauth nullok

# Prevent fingerprint prompts in non-tty background sudo processes
auth        [success=1 default=ignore] pam_succeed_if.so service in sudo:su:su-l tty in :unknown
-auth       sufficient  pam_fprintd.so max-tries=3 timeout=8

# Arch's default fallback
auth        include     system-auth
account     include     system-auth
session     include     system-auth

The interactive flow goes like this:

[sudo] password for user:

If you want to enter a password, just type it. If you want to use your fingerprint, press Enter on an empty field, and it will switch to:

Place your right index finger on the fingerprint reader

This approach feels more intuitive in the terminal than "fingerprint first," but it has its own little quirks.

The Catch with Option B: Empty Enter Can Leave a Failure Record

That "empty Enter" step inherently goes through pam_unix.so first. From pam_unix's perspective, an empty string is not a correct password, so it registers an authentication failure before the PAM flow moves on to pam_fprintd.

When you check faillock, you might see a record like this:

user:
When                Type  Source       Valid
2026-05-08 ...      TTY   /dev/pts/1   V

This means that even if the subsequent fingerprint authentication is successful, that initial empty Enter step might still be logged as a failed attempt.

So, if you go with this option, keep a few things in mind:

Don't set faillock's failure threshold too low.
Always leave a working root backdoor while debugging PAM, like a currently functional pkexec.
After consecutive failures, remember to check:

faillock --user "$USER"

bash

And manually reset it if necessary:

faillock --user "$USER" --reset

bash

Also, a side note: if you want the most stable graphical privilege escalation experience, it's best to just prefer Polkit for daily use and not force everything through sudo.

KWallet: Why You Can't Unlock It with a Fingerprint

This is another common point of confusion.

The role of kwallet-pam is, when you log in with your password, to pass that login password to KWallet, allowing the wallet to be automatically unlocked with the same credentials.

There's a line in the logs that says it all:

pam_kwallet5(plasmalogin-autologin:auth): Empty or missing password, doing nothing

This line makes the problem crystal clear: an automatic or fingerprint-based login does not provide the plaintext "user password," so KWallet doesn't get the key it needs.

A fingerprint authentication only answers one question:

Is this person who they claim to be?

KWallet's decryption requires an answer to a different question:

What is the key or password used to open the wallet?

The fingerprint device won't spit out your login password; it only tells the system if the authentication was successful. Therefore, KWallet has no way to decrypt using a fingerprint alone.

There are a few ways to handle this:

Disable automatic login, log in with your password, and keep your KWallet password synchronized with your login password.
Accept reality and manually enter your KWallet password once after reaching the desktop.
Reduce wallet security, for instance, by using a wallet with an empty password—but this is absolutely not recommended, especially if you're storing important keys.

So, this isn't a case of KDE failing to connect things; it's that "authentication" and "decryption" are fundamentally two different things.

`fprintd` Device Contention Issues

During my tinkering, I also encountered this error:

Device was already claimed
Verification was in progress, stopping it

This usually means a previous verification didn't terminate cleanly, or several authentication requests are simultaneously trying to claim the same fingerprint device.

The fix is simple:

systemctl restart fprintd

bash

If you lack sufficient user permissions, use your confirmed-working Polkit backdoor:

pkexec systemctl restart fprintd

bash

This is also why I don't recommend configuring too many authentication entry points to automatically grab the fingerprint reader. The device can serve only one verification request at a time. When multiple PAM or Polkit requests compete for it, the resulting experience becomes incredibly erratic.

Final State Summary

After all that tinkering, here's the current fingerprint and authentication configuration for this machine:

fprintd: Correctly identifies the Goodix device. I have enrolled my right index finger.
KDE User Settings: Fingerprint can be managed through kcm_users.
Polkit: Prioritizes fingerprint, falls back to password on failure.
kio-admin: Installed, allowing graphical root file operations via the admin:// protocol.
sudo: Asks for password first; pressing Enter on empty triggers a fingerprint attempt (max 3 tries).
Plasma Login Manager: PAM is configured for fingerprint, but auto-login bypasses it.
KWallet: Cannot be unlocked with a fingerprint alone.

Conclusion

The Linux compatibility of this Xiaomi Book Pro 14 is far better than I imagined. The few issues I faced all had clear causes and solutions, and in general, it's been a very smooth experience.

Another feeling I have is that tinkering with systems is much faster than it used to be. In the past, I'd have to dig through piles of ArchWiki entries and man pages for a problem. Now, you can just ask ChatGPT "how to fix xxx," and the efficiency is on a whole different level. This time, I even used OpenAI Codex, making the overall debugging process smoother by orders of magnitude.

mizore / notes

gdirect: Direct Google Drive Access from the Terminal — How It Works and Setup Guide

TL;DR: How Do We Connect Directly Without a Proxy?

Installation

Option 1: One-Line Install Script

Option 2: Homebrew (macOS / Linux)

One-Time Google Cloud Setup Guide (~10 Minutes)

1. Create a Project and Enable the Drive API

2. Configure the OAuth App

3. Add the Drive Scope

4. Create a Desktop Client and Download the JSON

5. Allow Your Account to Sign In (and Avoid the 7-Day Expiry)

6. Log In from the Terminal

Mainland China Network and Proxy Configuration Guide

Approach 1 (Recommended): Rule Mode + System Proxy, TUN Off

With TUN On: How Does a Proxy Hijack Direct Traffic?

Approach 2: When TUN Must Stay On, Keep Transfers Away from Sniffing and IP Rules

Daily Usage & Troubleshooting Reference

Common Commands

Troubleshooting Quick Reference

Correction and Improvements: 183.56.143.147 Is Not Google

What the First Version Got Wrong

Check 1: Who Does Google See as the Client?

Check 2: Can the Genuine Google Nodes in China Serve Drive?

Check 3: Other Intuitive Candidates

Check 4: Sweeping Google's Own Front Ends

A Note on Terminology: What GFE Is

What Changed in gdirect 0.7.0 / 0.8.0

Why We Don't Bundle Reverse Proxies Found Online

Limits

Technical Deep Dive: How Is It Implemented Under the Hood?

1. Where Candidates Come From: Google's Own Address Space and the Sweep Order

2. A Two-Stage Sweep: TCP First, Then Full Verification

3. Forged-Identity Fallback and Endpoint Failover

4. Custom libaria2: Why Patching the C++ Source Was Necessary

(1) Allowlist of Google Drive Transfer Domains

(2) Intercepting getaddrinfo and TLS SNI in SocketCore.cc

(3) Swapping Identities in Request.cc: Connect as g.cn, Real Host in the Host Header

(4) Stripping Authorization Across Redirects & Redirect Allowlisting

(5) Removing the OpenSSL 3 Legacy Provider Dependency

5. Certificate Verification Trusts Only Google's Own GTS Roots

6. Overwrite Guards, Virus-Scan Confirmation Pages, and MD5 Verification

7. 32 MiB Resumable Chunked Uploads

8. Localhost Loopback + PKCE OAuth Flow

Putting Memoh Under the Knife: Specializing an Agent for Cheaper Telegram Group Chats

Where Does the Money Actually Go?

Change 1: Clock Out Right After Sending

Change 2: Send Text, Reply, and Sticker in a Single Call

Change 3: Move Everything That Changes to the Tail of the Prefix

Change 4: Invisible Request Slimming

Change 5: Giving Text-Only Models "Eyes"

Bonus Telegram-Specific Polish

How Much Did It Actually Save?

Closing Thoughts

New Blog Features: Language Icons, Source Quotes, Tickers, and Charts

What's New

Language Icons on Code Blocks

Restyled Inline Code

Quoting GitHub Code Directly

Line-by-Line Diff Highlighting Like GitHub

Single-Line Tickers Become Mini Charts

Shortcode for Advanced Charts

Rendering ECharts Directly from Code Blocks

How It Works Under the Hood

Turning Shortcodes into Components in Remark

Loading TradingView Only on the Client

Enhancing Code Block Title Bars in Rehype

Keeping Inline Code from Stealing Attention

GitHub Code and Diffs Still Go Through Shiki

The Mental Model for Writing Posts

Fixing Obsidian Web Clipper Not Working on KDE Plasma Wayland

Cause

Analysis

Solution

Xiaomi Book Pro 14 on Linux

Problem 1: Screen Artifacts

Problem 2: Keyboard Not Recognized

Fingerprint: The Main Event

Why doesn't sudo pop up a fingerprint GUI like macOS?

A Key Limitation of pam_fprintd: The PAM Stack is Serial

4. Custom `libaria2`: Why Patching the C++ Source Was Necessary

(2) Intercepting `getaddrinfo` and TLS SNI in `SocketCore.cc`

(3) Swapping Identities in `Request.cc`: Connect as `g.cn`, Real Host in the `Host` Header

(4) Stripping `Authorization` Across Redirects & Redirect Allowlisting

Why doesn't `sudo` pop up a fingerprint GUI like macOS?

A Key Limitation of `pam_fprintd`: The PAM Stack is Serial

`fprintd` Device Contention Issues