I watched "Citizenfour"(imdb.com/title/tt4044364/) documentary and at
41:12 Edward Snowden gives a brief overview of some of the leaked
documents to journalists Glenn Greenwald and Ewen MacAskill. At 42:57
Snowden mentions devices which are able to collect data at rate of
1Tbps. This was in 2011. Screen-shots from the movie can be seen here: https://nsa.gov1.info/dni/2014/tumult.jpg Third slide looks like some
sort of vendor product roadmap
Just out of curiosity, what kind of equipment those might be? Is it
realistic that NSA/DoD are able to produce their own hardware? Let
alone custom silicon like Cisco or Juniper are. Or do they use off the
self hardware.. In addition, it's relatively easy to install a passive
fiber optical tap for a submarine cable, but how do you get
information out of it? I mean all the different wavelengths(CWDM/DWDM)
within the same cable, line rates(up to 100GigE), circuit switched and
packet switched technologies which those devices should support.. In
addition, how(bandwidth and network wise) to transport this data to
data analysis and storage equipment if it collected far away from
USA..
Some of those questions or thoughts might be naive and stupid, but
that's what crossed my mind when I watched the documentary. Maybe
somebody, who has done more research in this field, could clarify.
I see, thanks! However, this all requires at least some level of
Internet operator cooperation? For example if ISP in Northern Europe
owns a sub-marine cable between Finland and Sweden and they decide to
upgrade their legacy Nortel equipment with STM-64 line-card in both
ends of the cable to a Juniper T1600 core routers with 100GigE
line-cards, then it's not possible that intelligence agency equipment
supports this, is it?
In addition, how is the collected data transported for storing in
(NSA) datacenters and later analysis? I guess the data collection
actually has to be fairly selective simply because the amount of data
is so huge. For example take the large Internet Exchanges where
several Tbps of data are exchanged in peak hours each day.
NSA has had in-house chip fab facilities for at least 10 years, probably
closer to 20, and possibly as much as 30, as well as working agreements with
big network gear manufacturers.
It seems they had a system to backup 3 days of the internet, all data.
But such system failed because Internet generated too much data. So
Turmoil is a programmable event based filter, detect events and when
the event is triggered, save data from the stream. So they generate
as much data they want or can handle.
