Surveillance nation.

It may seem that the government keeping an eye on every bit of data flowing across the Internet is an improbably vast form of surveillance, too expensive to manage. Ars Technica informs us that it is terrifyingly easy to nose around inside all our emails, chats and site visits, using a series of functions that include deep packet inspection (DPI). DPI is hardware capability that has been used by no less than that paragon of democracy, the Libyan government under Muammar Gaddafi.

Deep packet inspection is useful because it keeps networks safe. However, it can also reveal the entirety of a web user's digital trail. If your data flashing through your Internet provider's routers is like a car going through a stoplight, data packet inspection is performing the function of the traffic cam that captures your plate number. But when used for snooping, data packet inspection doesn't just snapshot a random packet, it works full-time. This is why DPI's usefulness in probing data was feared by opponents of the Stop Online Piracy Act (SOPA).

As Ars Technica's Sean Gallagher reports, however, deep packet inspection is the tip of the iceberg when it comes to total data surveillance. There are services, Gallagher writes, that offer "Google-sized surveillance":

NetFalcon launched as a product just over a year ago. It uses a columnar database format similar to Google’s BigTable and Teradata’s Aster database systems as its data store, and can perform both real-time and after-the-fact analysis on data picked up by its network probes. Each probe can handle up to 10 gigabits per second, and the "correlation engine" that takes in all of the inputs can pull in over 100 gigabits per second for processing. NetFalcon’s “retention server” database takes inputs not only from the system’s network probes, but also pulls in feeds from external log sources, Simple Network Management Protocol “trap” events, and other databases. It correlates all the traffic and event data for weeks or even months. "Hundreds of terabytes or petabytes of data, but laid out in such a way that you can do queries and searches very rapidly," [Bivio Networks CEO Dr. Elan] Amir said.

If you are imagining a kind of covert spy Google accessible only to those who feel they have reason to track what you do on the Internet for their own commercial or just plain nefarious reasons, that seems about right.

Sleep well.