On September 14 Joseph Cox at 404 Media published the Project Lily documents. Hundreds of contractors read real ChatGPT conversations, summarize the user’s intent, and score four model replies from 1 to 7. Superpower Daily ran it the same afternoon. Today @HedgieMarkets and @AITrailblazerQ were still walking the default toggle. OpenAI’s help page is the source for the control. “Improve the model for everyone” is on for Free, Plus, and Pro. Enterprise, Business, and Edu start off. Turning it off applies to new chats. It does not pull back what already left.
I wrote on the 15th that a self-hosted sandbox does not move the session. This is the consumer version of the same hole. You can hide the username. You cannot hide the file from the person paid to grade it.
What the dashboard shows
The work is not the safety queue. 404 Media is careful about that. Flagged harm is a different pipeline, the one OpenAI already talks about in public. Project Lily is style and usefulness. Reviewers are told to punish sycophancy, emoji clutter, “AI-speak,” and any line that pretends the model has a life. “As a chef, I like to” is misaligned. “I’ll take a look” is fine. They do not have to fact-check with a search. They do have to mark medical, legal, and financial answers that skip sources.
The dashboard does not show the ChatGPT username. Some tasks show a “user memories summary” above the prompt. 404 Media saw overviews of what the person had used the bot for, and in some cases where they might live. OpenAI told the paper the chats go through Privacy Filter first. The product page for that model says it can miss uncommon identifiers and ambiguous private references, and that it can over- or under-redact when the context is short.
That is not a fence. That is another model.
A reviewer in North America told 404 Media they are paid more than $50 an hour. The recruiting firm is Crossing Hurdles. Pay comes through Mercor. Meta paused Mercor in April after a breach, according to WIRED, which 404 Media cites. The worker said they do not think users know a contractor is analyzing the conversation. Some of the prompts 404 Media saw asked ChatGPT to keep the content to itself.
After the paper asked where users are told that humans may read prompts to improve replies, OpenAI did not answer. It updated the help page on the training toggle. The page still does not say a person may read the chat.
ChatGPT had more than 900 million weekly users as of February, the number 404 Media used from TechCrunch. The training setting is on unless you find Data Controls and flip it. The worker found the job through a staffing firm whose site says human intelligence powers AI progress. The model is graded by people looking at your thread.
What the toggle is not
The setting only covers chats you send after you flip it. Off means new conversations stay out of model improvement. OpenAI’s privacy policy still says deleted chats can remain if they were already de-identified and dissociated for training. Abuse and safety review is a separate door. OpenAI says authorized staff and vendors can still see content for investigations, support, and legal matters even when training is off.
Anthropic told 404 Media it also uses human review when “Help improve our AI models” is on, after stripping account identifiers. Google’s Gemini disclaimer already says humans review some saved chats. Labs have done this for years. The consumer default is the part that matters, and the memories block riding along with the prompt.
Michal Luria at the Center for Democracy & Technology told 404 Media the chat window creates a false sense of a one-on-one. That is the product. A blank box is not a room.
I do not need a theory of RLHF to use this. If a stranger can be handed the thread, the thread was never private. Anonymized is a claim about the account name. It is not a claim about the text.
A useful answer can still score low if it is too long or cluttered. The confidential guide 404 Media saw says the model should generally match the user’s tone, but slightly less intensely, and stay professional without implying it is human. That is editorial work on a chat you typed because the box looked empty.
Keep the chat off their disk
The useful control is not a buried toggle that does not work backwards. It is whether the conversation is a file you can delete on a machine you own.
Local models still do not give you Astra. They do give you a session that never enters Crossing Hurdles’ queue. If the work is a diary, a medical question, or a Slack rewrite you would not paste to a contractor, the box has to be yours. ChatGPT Free, Plus, and Pro are a product that starts with a copy.
The filter is a model. The reviewer is a person. The opt-out is not a fence.