This seems to be a bit of too much ado over nothing. This type of projects have been going on for over a decade now (just search for Twitter or Facebook in Google Scholar). Even the government has been involved in financing some of this research (For example: https://www.iarpa.gov/images/press/osi/24_open_source_indica...).
Many interesting things can be done with this kind of data (and it's what originally got me interested in data science https://www.springer.com/us/book/9783319140100). That being said, the specific solicitation (https://www.neco.navy.mil/synopsis/detail.aspx?id=537809) doesn't seem particularly exciting or novel. Just an overgrown grad school project to keep the students busy. With some Python experience and patience to collect the data, pretty much anyone can do something similar.
Is there something recent that brought ‘Overton window’ into common use? I honestly believe the first time I’ve ever seen the term was within the past few months and now I see it regularly.
The reality is that private entities and external actors are already using data to figure out better offensive and defensive maneuvers. If the entities of your nation do not take similar steps, they are likely to be ill prepared to "serve & protect" you and your interests, which may include a fair and functional democracy.
Some may ask "why is the military involved" the answer relates to the fact that today's military have realized that foreign actors will weaponize technology (e.g. internet, social media) to serve their goals. To ensure that the goals of your nation are protected, the military has moved more and more towards the cyberspace.
Here are a few benefits that can come from such endeavor:
* Relief efforts - How do people react in social media before, at the time and after a disaster happens? How can social media be used to connect to the ones in need? How can X entity use social media to advise groups in a risky area (e.g. landslide) to diminish damage and risk of life?
* Trolls - Are there cues that can be used to determine when someone is a troll? Can that information be used to help society become more resilient in the future?
* Training - Can this research help dozens/hundreds of personnel to gain valuable know-hows and capabilities in the cyber arena? Will that knowledge translate in a more capable entity?
Another question to ask is whether X entity follows your ideological view and whether there is enough oversight. If the answer is no to either/or, then as part of a democracy, you should raise your concern with representatives & the people around you.
> Our research aims to provide enhanced understanding of fundamental social dynamics, to model the evolution of linguistic communities, and emerging modes of collective expression, over time and across countries ...
> As a central requirement for this research, we seek to acquire a large-scale global historical archive of social media data, providing the full text of all public social media posts, across all countries and languages covered by the social media platform. We aim to use this research to advance knowledge through scientific publications. This data will also be used for pedagogical purposes in the classroom, giving students new opportunities for thesis research and the development of “big data” analytic skills.
How can the Department of Defense do a better job at interfacing with the general public to better explain intentions, motives etc... with the work that we do?
They could start with being honest that those aren't the DoD's only motive behind wanting such a corpus and analytical model.
There are many positive and negative things that could result from something like this, and it isn't the fact that the Navy is doing it that scares people. It's the fact that the US government is doing it.
They could start with being honest that those aren't the DoD's only motive behind wanting such a corpus and analytical model
That would be akin to asking [insert company] if it will possibly ever use data collected from one thing on a different project at some later date. That is a complete unknown, so it wouldn't even be appropriate to put into a proposal, as they have specific formats.
NPS is a graduate study program, with a research component and long history of academic publishing. Whether that data could possibly, someday be used beyond that is completely out of scope for this solicitation. Further, if other groups in the military wanted similar data, they would likely not go to NPS for it, for a variety of reasons.
I think at the end of the day though, if someone has an assumption that everything the DoD does or messages to the world is somehow subterfuge, there's no amount of discussion that would reverse that feeling.
Ignoring your false dichotomy, it should be fairly obvious what some of the negative consequences of such an analysis of a large corpus of social data could provide for the US government, a totalitarian state with a long history of infiltrating and manipulating social movements.
I'm also a realist and understand that this kind of analysis is inevitable and will eventually be continuous. I'm not arguing that it shouldn't be done.
I remember being a uni student at a career fair nearly a decade ago and the folks at the CIA booth were wanting to hire to "predict the next Arab Spring" (which at that time was fully underway). When I asked a little bit, they didn't divulge more and weren't interested in me (my degree is not comp sci).
I've just been assuming this sort of work has been going on since then.
The idea of the military arm of the U.S. government predicting the next Arab Spring reminds me of that classic mind-bender of a quote, which I think is often misunderstood as a defense of lying to the public, when it is about something deeper:
> The aide said that guys like me were "in what we call the reality-based community," which he defined as people who "believe that solutions emerge from your judicious study of discernible reality." I nodded and murmured something about enlightenment principles and empiricism. He cut me off. "That's not the way the world really works anymore," he continued. "We're an empire now, and when we act, we create our own reality. And while you're studying that reality -- judiciously, as you will -- we'll act again, creating other new realities, which you can study too, and that's how things will sort out. We're history's actors . . . and you, all of you, will be left to just study what we do."
Have you heard of the free energy principle? The two ideas seem related.
From Wikipedia:
> The free energy principle is that systems—those that are defined by their enclosure in a Markov blanket—try to minimize the difference between their model of the world and their sense and associated perception. This difference can be described as "surprise" and is minimized by continuous correction of the world model of the system. As such, the principle is based on the Bayesian idea of the brain as an “inference engine.” [Karl] Friston added a second way to minimization: action. By actively changing the world into the expected state, systems can also minimize the free energy of the system. Friston assumes this to be the principle of all biological reaction.
Right, thats about when I also heard pitches about the same. Perhaps social media has changed now. I certainly use it differently. Or perhaps the US Military takes a decade to get research like this going?
Probably it's ongoing in defense research, just usually not much comes of it and it's not published a lot. I did something tangentially like this if you squint at it the right way, maybe 10-12 years ago, for some defense research grant. It would shock me if they didn't have a dozen other parallel grants going with other companies at the same time, and more since then.
If they are using public posts, this seems fine. It’s more of a general study and could have a real impact on our understanding of how stories spread (at least via this medium, today).
What is not fine is the fact that social media companies have spent years using dark patterns and tricks to get us to publish more and more of our info publicly. It used to be common knowledge that you never post any of your real life details on the internet but now the most popular websites will ban you for not using your real name.
I remember a decade ago when http://pleaserobme.com/ came out and how my coworkers were shocked. I honestly thought over-sharing was a common sense bad idea, but it was not as common sense as I had thought. They did not require any dark patterns to willfully give up everything to social media.
> It used to be common knowledge that you never post any of your real life details on the internet
But before then it used to be common knowledge that you always post your real life details on the internet, at the very least your name and email, often your phone and office address too
The real concern isn't the privacy given that it is publically available but what it says about their motivations and respect for civilian control of the military.
Monitoring pacifist groups based on publicly available information may respect rights technically but it is not a good look. It indicates either major bad intentions to subvert democracy or sets the stage for any who plan to abuse it while being peripheral to their nominal core functionality.
To give a crude analogy it may be technically legal to note the comings and goings of my neighbor and every aspect of their home security but it would be very understandable if they think I plan to burgle them.
I just assumed this was going on already. Maybe because they are working with a professor on this effort it had to be revealed. I wonder if behind the scenes the Navy or the military had already done this multiple times and this is just another attempt.
Or by announcing it, they give the impression that it is something they have not done previously. Indeed, they might want to see how some people react to this and a sudden stream of deletions of social media posts by individuals/groups after finding out about this - would be of interest too them.
Another possibility is that the military like a large company has many arms and legs and highly possible that it is something that is already done, but by some seperate part of that collective that the rest are not aware of. I've seen a company the size of 100 have two people do the same project for nearly 6 months before they realised about the other and the duplication of work. More people, more separate divisions/locations, more chance duplication on some levels happens.
For me, I suspect the later, but equally mindful that as a tactic - doing something and then after you have done it, say your going to do it to gauge reactions - is a tactic that would not be out of place in any military intelligence collective.
Yes, my point is that the military usually don't show what they can do and more importantly what they can't do. Should they have the best algorithm in the world to match anything with anything in that data, there would be many reasons not to reveal that.
I'm not sure there's been any evidence that the military has had success utilizing current computational approaches to data. E.g. the DCGS project was a failure and I don't think that project tried anything beyond what you'd get out of Microsoft Excel (https://nypost.com/2014/10/27/army-spent-5b-on-failed-techno...). DoD plans for projects that last for decades and the algorithms people are building for commercial purposes today are developing way faster than DoD's procurement can keep up I'd guess,
> "...and individual users won’t be identified..."
Perhaps not as individuals, with enough personal meta data your identity can be reverse engineered.
That aside, anyone you believes there is no hidden agenda here is naive. Any understanding will have a second benefit for doing the opposite (i.e., preventing movements).
> When is a movement and movement, and not a guerilla terrorist cell network?
That depends on who wins the war, and thus writes the history books. From the point of view of the British, the American Revolution was an act of terrorism and treason.
Attempts by the military to understand (and, one would assume, prevent) popular uprisings would nip a second American Revolution in the bud, for better or worse.
The linked article refers to an attempt to entrench the status quo, and not everyone is pleased about that.
It is no secret that foreign actors (including Russian and Chinese military intelligence) are employing “active measures” to influence political changes in Western countries. Additionally, information operations are now a part or modern warfare, in much of the current conflicts.
The US military is interested in mapping these activities and developing countermeasures.
>...information operations are now a part or modern warfare...
Information/disinformation operations have been a part of warfare since... ...forever it seems. It is - certainly - nothing new. The medium has merely changed and that's what important to the staging of the front[s], as it were.
You should pick up a copy of Eric Frank Russell's WASP[0], if you haven't read it before. It's a very good read in this area.
It is their main business. It has always been the first step before war against other country to support revolts and popular movements.
Sometimes, countries protect themselves using revolution in other countries. For example Lenin wanted to make the revolution in Germany, as they believed Russia was not prepared for revolution. Lenin was captured in Germany and released with the condition of not trying it again on Germany(or he would be shooted and killed) and sent to Russia, where he started the Soviet revolution.
For example, United States wanted to control the Panama channel, so they supported an "independence" movement against Colombia, so a new "free" country was created, that by "coincidence" gave the US the total control of it for a long time. Even today, the US is in control of it.
Before that, when the US felt the Spanish Empire was weak, they supported the "Cuba(and Philippines) libre" movement. People in Cuba and Philippines were very happy for the help until they realized the helpers army did not go away after the "independence". In Philippines the US exterminated over a million people, forcing everybody to speak English.
After Panama, the US continued supporting coups in other America's countries and around the world.
One of the most important movements the US supported was the popular movement in Persia, now Iran, because they wanted the British Empire to leave(so their own companies could enter and control the oil).
In the end, this movement converted Iran from secular country to a radical Islam one, and the Americans were ousted.
For the US it is routine creating havoc around the world, as the hegemonic power in the world that they are. All empires before had done that too.
They would love to be able to support popular movements in China or Russia as it is a threat for US hegemonic power in the future. Getting united China to disintegrate in several smaller countries would be a great victory.
Social media is at the very core of 4th generation warfare. Social media is more powerful than nukes because he who controls the discourse of the public is more powerful than he who destroys the public. He who controls the social network can make people believe whatever he wants them to believe to achieve his goals. Vastly more powerful, invisible, and humane than nuclear weapons, especially in terms of reach. What we are already witnessing online is an information war which hijacks and disrupts our social connections and thought processes to replace them with highly engineered stimuli which steer the people to think and act in a certain way. Neural networks are in development which will provide a positive feedback loop to continually evolve puppet account discourse in a way which maximally achieves the goals of a behavioral campaign. Phrases which maximize psychological impact for each campaign will be identified and automatically evolved based on replies and related discourse to always maintain maximum engagement as the overton windows shifts. These experiments are already well underway.
Is your comment meant to be sarcastic? If not, can you elaborate on why - of all possible options for the department who's primary purview in practice (in contemporary times) is the destabilization and/or destruction of other nations, you think this would be the most probable explanation for a mass scale analysis of 350 billion social media posts?
If that message comes off abrasive it's only because I am genuinely confused, but would like to know where you're coming from. There's always a risk of us living in our little bubbles and so I make every effort to try to see things through the eyes of others, especially when I simply cannot otherwise empathize with their views on any level.
I wasn't being sarcastic, they care a lot about situational awareness so have an interest in 'checking the weather' local to their operations, whatever those may be.
Government aid is a proxy for soft power through population pacification, political subversion, intelligence gathering operations, and proselytization. They know exactly who they want to target at all times.
As always, propaganda. For instance we recently made made major efforts to try to overthrow the government of Venezuela that included, directly or indirectly, digital propaganda efforts. These efforts are ongoing but seem to have failed. Why? It can also be used internally. For instance in the 60s see things such as COINTELPRO [1] or the FBI's efforts to attempts to blackmail and push MLK to suicide. [2]. COINTELPRO is probably where you can draw the most analogs relevant to today - planting fake stories in the media, subverting groups through internal infiltration, compelling 'private' media to prohibit access to targeted groups, character assassination, etc. What are the effects of these actions on digital media, and how can it be used to pursue our interests? The same is true of MLK. The FBI's primary goal was simply to destroy him as a leader. What causes leaders of online movements to disappear in ways that help splinter groups rather than turning them into martyrs?
Bigger question is why in the world this would be made public. I think it's a 100% safe bet to say that the military/intelligence has been engaged in this research for many years already. I see no positive benefit to making this information public aside from preempting 'leaks' / creating preemptive PR.
Should you expect & accept for everything you speak in public to be recorded and monitored? Is it "fair game" to do facial recognition on everybody that shows their face "in public"?
You're "tweeting" to your followers and random people that happen upon your feed can read it too, just as some random person sitting on a bench in the park would hear you speak. I don't believe that most people (normal people, that is, not professional pundits that use Twitter) are fully aware that they are "completely public" most of the time. They'll likely "know" if you ask them, but they'll use Twitter as if it's an extended group chat a lot of the time: they don't expect that the world is watching/can watch.
There's probably a cultural component to what side of this one comes down to, how much privacy you can expect (and be granted by law) in public spaces is different between countries.
When I'm done speaking, I'm done speaking. The moment has passed. I know the information is now safe unless my friends share it.
I don't have to worry about someone seeing a crude joke made 10 years ago and starting a massive deplatformization campaign against me in the name of social justice.
When I make a post, it's there forever.
I know your analogy seems to make sense, but these two phenomenon are incompatible. You cannot translate this digital phenomenon to that physical phenomenon.
> You cannot translate this digital phenomenon to that physical phenomenon.
I agree, they aren't the same in that regard. The expectation is similar, that's what I was trying to say. When people make a crude joke or call somebody an asshole on Twitter, they don't expect that to be read by everybody.
Somebody compared a public post to standing on a soap box. If you're standing on a soap box, you expect what you say to be heard publicly. I believe that's primarily because you usually don't stand on a soap box, so being that exposed really stands out. If everybody stood on a soap box all the time because the floor is lava and soap boxes are heat resistant, that would quickly change and people wouldn't associate a soap box with "public". The early days of social media certainly had that, it was new and everybody was acutely aware of it. That changes with exposure, now it's no longer an outlet to let the public know about your opinion, it's a chat where you talk one on one or in small groups (for most users, very few have large followings and get a lot of visibility) to friends and strangers alike. Having some tweet go viral is the equivalent of some reporter overhearing your private conversation and mentioning what a vile person you are in his column.
> When people make a crude joke or call somebody an asshole on Twitter, they don't expect that to be read by everybody.
If we're keeping up with analogies, that's like saying someone doesn't expect what they say on a soap box to be heard by anyone. That's a false expectation, however.
We have to strike a balance between ideology and pragmatism. The best compromise is likely preventing the government from having access to data we consider private, while understanding anything on the other side of the line is fair game and accessible to friends, family, politicians, corporations, and historians alike.
The only way we can achieve this is by limiting what can be done with this data aka fixing the US government. We cannot, for example, have public or inferred health data used to prevent people from attaining health insurance.
> If we're keeping up with analogies, that's like saying someone doesn't expect what they say on a soap box to be heard by anyone. That's a false expectation, however.
That's my point. If "soap box speaking" was as common as tweeting or posting on Facebook or speaking to a friend in a pub, people wouldn't expect it to be different. When social media changed from "technology pioneers use it carefully" to "everybody uses it and rarely thinks twice before hitting send", the expectation of the average user shifts.
> while understanding anything on the other side of the line is fair game
I do agree: Fair game for any person. As soon as technology enters the game (scrapers, databases etc), it changes. Similarly, you can sit in a park and catch fragments of what the people walking by talk about, that's all right. If you're using a directional microphone to listen in on people on the other side of the park, that's not. A user doesn't expect to the target of a microphone or a webscraper/data mining operation.
> Fair game means fair game. It means its open for anyone.
In that case, you're allowed to remotely listen in on everyone as long as they're in a public space.
> If people don't expect this, it's because they don't understand it.
Absolutely, and because everything is done so that they either don't understand it, or quickly forget it. Change Twitter's send button to "I want everyone, including my enemies, the government, journalists and my mother to read this" ... and you'll pretty much kill Twitter.
> Let's work on educating people instead of warping reality around the ignorant.
The ignorant make up the majority. We can't educate everybody about machine learning to the required degree where they can make an informed decision about it. The same goes for financial instruments or law. Instead, we just outlaw the most egregious transgressions ("no, it said on page 277 of the contract that we get all the money he'll ever make in return for this bicycle") and base what's allowed on what a reasonable person would expect.
Making a public post is like standing on a soap box in the park. If you and I go to the pub and I tell you that the Mayor is a crook, then that's between us. If I stand on a soap box and shout that the Mayor is a crook then that's slander and I should either have to prove it or I should be sanctioned.
And yet, once the technology is mature enough (it may already be, just not that cheap too deploy yet), I'm sure the government will integrate CCTV cameras with microphones that can listen in on private conversations between individuals and analyze that in real-time.
I'm also sure the government will shamelessly make the argument that because you're telling something to a friend in a "public place" it means it's not a private conversation, so it's fair game for mass-collection and real-time analysis. It may have already made this argument in some court cases, but I don't recall exactly.
Similarly, your private Facebook posts are already compromised.
What the court does is muddy the distinction between a theoretical reasonable expectation of privacy and the actual expectation of privacy resulting from how things currently are.
As in, I expect conversations in my car to be private, but if all cars had microphones with cell connections my government would argue that I can't expect privacy because clearly my environment already has privacy-compromising features. [0]
It's cyclical and deceitful and until we stop playing ball and change the language, the farce will continue.
To me the issue is proportionality and scale - for example if one person a year out of 100M is suspected of plotting to introduce a killer virus to the general population then (provided the suspicion is reasonable based on evidence) I feel that very few people would object to this kind of surveillance. No more than they would object to the idea that you might be talking directly to someone who has decided to become a federal witness in order to stop you.
On the other hand if the government turns on all connections and starts to bulk process all of them to find "interesting" discussions, and then complies dossiers based on these which are then used to determine probabilities of behaviours... and issue sanctions, constraints and awards based on these.. Well. My bet is that almost everyone would object.
Somewhere between these two extremes, absolute surveillance and extremely limited state power lies the correct position. My own view is that it should be much closer to the limited position than state omniscience, the core question is do you want freedom, or do you want safety. In the modern world you (I, anyone) can't have an absolute of both. Absolute freedom would result in the dominance of lunatics with assault rifles (cf. anywhere in the modern world where state power collapses). Absolute safety means that your neighbours and family will regularly face state sanction - even if by some chance you don't.
The fly in the ointment is that I fear that state omniscience is stable in that a society run in that way could continue indefinitely. On the other hand the extreme of liberty (given assault rifles) is clearly not sustainable (in that school, hospital, roads, power, water, cannot be delivered). So we may see a world in which a core territory (China) is run by big brother and a periphery (to steal William Gibson's term) is allowed to seethe and boil in anarchy.
To avoid this we need to develop civil society and generally learn to get along with each other, and we need to punctuate this with action to lock up lunatics and psychos who are taking advantage of us. We also need to find a way to neutralise a core that is seeking to actively destabilise the periphery. Interestingly I think that HN is one of the very few places where this kind of action can be developed.
This is legitimate if the people being watched have bona fide suspicion of criminal intent, and if the extent of surveillance is proportionate to the social threat.
> In a perfect world people would use this as an opportunity to tell the government to shove it
In your vision of a perfect world. Personally, I’m fine with this. They aren’t using call logs or other private data. Information on social media is public or semi-public. Fair game, and I’m curious about the results.
(Agree that a more productive public debate is needed. But that shouldn’t start by presuming values.)
The problem in this case is not that private data might have been used. The problem is that the outcome of this research will be the US military developing a weapon that can be used to create or stop or influence political movements across the world (including inside the US). The question is whether it is a good idea to create such a weapon. War is inevitable but ever since time immemorial people have respected certain rules of engagement. Its difficult to draw the line in the sand to decide what type of propaganda is unacceptable, but it should be obvious that propaganda created using every dark pattern in the book, trained on 350B social media posts is probably not going to bode well for Earth's future. Its a lose-lose situation.
> whether it is a good idea to create such a weapon
These “should we do X” questions tend to be useless. If it can be done, it will be done, absent multilateral action. Being able to predict social upheaval for humanitarian and geopolitical positioning purposes is obviously useful.
The government using such models to suppress activism in the U.S. is bad. But knowing how such phenomenon work is also a precursor to preventing its abuse.
So you're okay with someone following you 24/7/365 every moment that you step outside your house? By definition, this type of surveillance data would still be public information.
> you're okay with someone following you 24/7/365 every moment that you step outside your house?
One problem with the current state of the privacy debate is this sort of hyperbole.
No, I’m not. I delineate between information people intentionally cast publicly—even if they don’t understand the consequences of doing so—and information that occurs in public. My location, tracked by my phone and router to various apps’ servers, occurs in public. My tweets, on the other hand, are intentionally broadcast. There is a lot of grey area between these poles, but intent broadly inspires my thinking.
It isn’t hyperbole. Your existence outside of your private home is “occurring in public.” This is why the concept of paparazzi exists. The only difference is that normal people don’t have an army of photographers following them.
Extending “I’m fine with the U.S. military analysing social-media posts” to “okay with someone following you 24/7/365 every moment that you step outside your house” is hyperbolic. Framed as a question, on the other hand, it invites discussion.
I would be very surprised if this hasn't been going on for years already, back in 2014 they were already as far as using Facebook users as test-groups [0].
Then there was also the case of a Pentagon contractor leaving their AWS buckets open exposing billions of scraped social media posts. [1]
As much as people can reach for benign reasons for "research" like this, the reality is that it's not solely done to "understand" movements but rather to better influence them [2].
Many interesting things can be done with this kind of data (and it's what originally got me interested in data science https://www.springer.com/us/book/9783319140100). That being said, the specific solicitation (https://www.neco.navy.mil/synopsis/detail.aspx?id=537809) doesn't seem particularly exciting or novel. Just an overgrown grad school project to keep the students busy. With some Python experience and patience to collect the data, pretty much anyone can do something similar.
You can find the slides for one of my old (PyData 2015) tutorials on social media mining here: https://www.slideshare.net/bgoncalves/mining-georeferenced-d...