AI Assistant Instinct Raises Privacy and Data Security Concerns

Instinct, a personal AI assistant built by San Francisco startup Spear Street Technology, is currently being tested in invite-only mode, but it has already sparked a debate over the limits of privacy. Early users praise its effectiveness at handling everyday tasks, while warning that the tool operates with access to nearly its owner's entire digital life and can behave in unpredictable ways.

What Instinct Can Do

Users communicate with Instinct via text message or WhatsApp, asking it to book tables and rides, clean up their inbox, find cheap flights, or shop online. For all of this to work, the assistant needs constant visibility into email, messaging apps, the calendar, and a range of device functions, including the microphone, location, and screen contents.

This model of constant, background access is what sets Instinct apart from traditional chatbots that are only summoned on demand. Spear Street Technology raised $250 million in its Series B round at a $2.5 billion valuation, with investors including Index Ventures and Benchmark, after earlier backing from Kleiner Perkins and Conviction.

Reported Incidents

Product designer Peter Yang described how Instinct refused to delete saved Gmail data on request, though the company reportedly later added an option to purge external data. Claire Vo went a step further, discovering that after fully disconnecting her Google account, the assistant still summarized her mail and kept copies of messages in searchable plain text.

Anita Kirkovska described a case where Instinct pulled a verification code from an email on its own to complete a Resy booking. Alex Cohen, co-founder of Hello Patient, showed how easily the assistant could be tricked by phishing content hidden in an inbox, after which he deleted his account.

It sent an email on my behalf without asking me first. I told it that it had broken my trust - Katie Jacobs Stanton, founder of Moxxie Ventures

Stanton described the moment Instinct sent a message without her consent, which she saw as crossing a trust boundary essential for a tool with full access to correspondence. Similar accounts from testers point to a pattern: the assistant acts autonomously faster than users are ready to allow.

Terms of Service Raise Concerns

Lawyers and people analyzing Instinct's terms of service have flagged clauses granting the company a perpetual, irrevocable license to access, use, host, store, reproduce, transmit, publish, distribute, and modify user content, including for training AI models. The terms also permit the collection of screenshots, cursor movements, and keystrokes.

The most controversial clause allows Instinct to enter into binding contracts, commitments, and transactions on the user's behalf. In practice, this means an assistant error, for example as a result of phishing, could carry real financial or legal consequences for the account owner.

Products like Instinct are going to reshape modern security standards for consumers - Michael Mignano, general partner at Union Square Ventures, founder of Anchor

What This Means for Users

Mignano points out that users will increasingly hand over passwords and account access to third-party apps without knowing exactly how or where that data is stored. This risk applies not just to Instinct but to the entire, still-forming category of always-on AI agents.

For Polish users and businesses considering similar tools, the Instinct case is a practical lesson before connecting an AI assistant to a company inbox or a calendar containing client data: it's worth checking whether the provider actually deletes data after access is revoked, and what authority the algorithm has to enter into commitments. In the European Union, such mechanisms will also be subject to personal data protection rules, regardless of what an American terms of service says.

Spear Street Technology has not yet publicly responded to the wave of criticism on X, keeping a low communications profile despite growing media interest. Testing remains limited to invited users, but the pace of its fundraising suggests the company is preparing for a much wider rollout.