I was surprised by the number of false positives reported here, went ahead and tested on several Twitter accounts of my friends, both professional and personal. 9 out of 25 tested were classified as 'bot' with probability > 0.6.
Only 11 were classified as 'humans' with probability > 0.7.
And that's on 25 accounts of people I know personally.
Given the preposterous error rate, I deem there is no actual classification logic in R, and instead it uses the (very fallible) humans to do the actual classification via a Mechanical Turk-style API.
Wait, do I need to add "/s" to this post, or is it obvious enough?
I've done some work in this area - it's disappointing how terribly the Twitter product has failed to evolve to take account of bot usage.
There are (of course) some useful bots, but lots of incredibly harmful bots, they they should be treated differently to actual humans.
But Twitter can't ship product, so it's not really worth suggesting what they should do.
In the mean time, my colleagues and I got a nice WWW18 conference paper about a new unsupervised (!) way of detecting some type of bots on Twitter. Like most things it's completely obvious in retrospect...
I think they have very little incentive to get rid of boys and fake accounts. Same as Facebook. It would reveal how many active users they really have.
I looked at the GitHub README for the project, which says
> Uses machine learning to classify Twitter accounts as bots or not bots. The default model is 93.53% accurate when classifying bots and 95.32% accurate when classifying non-bots. The fast model is 91.78% accurate when classifying bots and 92.61% accurate when classifying non-bots.
Overall, the default model is correct 93.8% of the time.
Overall, the fast model is correct 91.9% of the time.
How is this accuracy determined? There is no information available explaining how this determination is quantified, nor what the caveats are.
I agree this is probably what happened. From the estimation object I see around a total of 3500 training samples with around 90 predictors. 3-fold cross-validation was done so at each iteration only around 2150 samples were used for training... And it seems none of the samples were used for out-of-sample checking.
> The default [gradient boosted] model uses both users-level (bio, location, number of followers and friends, etc.) and tweets-level (number of hashtags, mentions, capital letters, etc. in a user's most recent 100 tweets) data to estimate the probability that users are bots.
Not an exact science, but shows what you can do and deploy quickly with R/Shiny.
The author’s rtweet package is very good for making quick Twitter data visualizations.
I tried putting in verified users and they were all "probably bots". By definition is that not the only type of user publicly acknowledged as "not a bot"?
While the quality of the model can be debated (I noted lots of false positives too), I do note that it’s kind of cool that we’re all sitting around and poking at an app written in an R web framework.
If you haven’t:
1. Downloaded RStudio IDE
2. Built a hello word Shiny App (better still for a flavor of the thing a hello world app using the shiny dashboard package)
3. Deployed your app to shinyapps.io
I highly encourage you to do so if for no reason than to see how streamlined RStudio has managed to make web app deployment for people who often don’t have much of a programming background.
I’m continually impressed with the work RStudio does, even if I’m a curmudgeon and still write all my code in Emacs instead of their IDE. If RStudio expanded to support Python similarly well, I imagine they could really be the place most data scientists work.
RStudio is by far my favorite IDE. I would write much more Python if there was something as good for it. Anticipating more Python support, which I believe is part of the plan
False positives are usually due to high number of retweets and. . . uh. . . check your followers. You may have a lot of bots following you or you may be following a lot of bots.
I recently watched a talk from 34c3 (chaos computer club conference) which were held at the end of last year about Twitter bots, their existence and their detection.
The speaker couldn't find a lot of bots that were cited in studies and that their methodology were somewhat arbitrary.
I believe there is a bit of confusion in the public discussion, mostly of the 2016 election:
While “bots” featured large in initial discussions, the details that have emerged since, especially in the recent indictment, actually point at software-assisted humans as the main avenue for these activities.
I believe the “bot” monicker because (a) it’s how most people would tackle the task, perhaps ignoring the cheap labor pool this shadowy PR operation had access to, and (b) the bot-like behavior exhibited by many of these accounts, which was actually a result of not being native speakers, following a script, and having no deeper knowledge of politics and culture to fall back on.
I'd opine the choice of model matters less than the general approach used. In this case, features are taken from the data that are more like metadata about an account, metadata about the frequency and such of tweets, and automatically generated text features about the tweet text which are used to train/predict the binary classifier. Whether it's gbm, lda, logistic regression, etc. used on this data is probably not that huge a deal (maybe small % differences in accuracy) assuming each are used correctly.
What I'd want to also see however, would be curation of a relevant corpus of tweet content that could be integrated in the classification process in terms of actual tweet content (bag of words, sentence structure, & other NLP features, images, other media) being used to determine a given account's similarity between bots and non-bots -- instead of relying essentially on metadata similarity in addition to crude auto-extracted text features like simple total number of words and character counts. I can't say for sure if it'd improve predictions but would seem key for this type of problem.
I tried a few things, it seemed to be working well but now I keep getting "An error has occurred. Check your logs or contact the app author for clarification."
@realdonaldtrump is only 0.008, interesting comparison! I wonder if it's picking up something weird from the style of @potus changing from Obama to Trump, and in particular something related to the fact Trump seems to favour his personal account even as POTUS.
Jacked? I just tried searching for "bortnot" and I can't even find your plugin in the search results. You can't blame somebody for stealing a name that nobody can even find.
Given the preposterous error rate, I deem there is no actual classification logic in R, and instead it uses the (very fallible) humans to do the actual classification via a Mechanical Turk-style API.
Wait, do I need to add "/s" to this post, or is it obvious enough?