"(Craigslist) said it offered a license that would have allowed PadMapper to use its data on mobile applications but that the competitor did not accept the terms."
But on the other hand, "users and developers are exasperated with Craigslist’s insistence on preserving an outdated interface and design."
I don't know the legal merits of either party's position, but from an ethical perspective, I tend to side with Craigslist here. Dissatisfaction with a commercial site's UI is not just cause for using their data without permission, particularly if they had made an effort to offer a licensing agreement, whatever the terms might have been.
I see this from the opposite viewpoint, that PadMapper is operating as a search engine designed to help you find the right Craigslist posts for your apartment hunt. Claiming that PadMapper's success is from Craigslist failing to 'provide a good UI' is like suggesting that Google News' success is because newspaper companies failed to provide good article SEO. Instead, it's a different modality of content discovery, and it ultimately redirects you to the original source.
In my (personal) opinion, I would argue that the content providers or aggregators and search engines benefit synergistically. CL and Belgian newspapers appear to disagree.
Here's the abstract for a nice review of search engine law: http://works.bepress.com/james_grimmelmann/13/ If I remember correctly, indexing a site that asks to not be indexed might be illegal as an illegal tresspass, but it is not settled law. The argument is that you are stealing resources (computer time) from the site owner.
There are a lot of companies that are scraping Google quite successfully. Many of these are for 'rank checking' services that provide ranking data for certain keywords over time; these are heavily used by SEO and marketing agencies.
The two that jump to mind are Authority Labs and SEOmoz.
IANAL but AFAIK it is only a civil matter (i.e. not illegal) since it is a usually prosecuted as a tort of trespass to chattels. For such a case to succeed the prosecution needs to show that the actions of the defendant deprived them of use of the good they were trespassing on. i.e. they need to cause enough of a burden on the servers that the claimant or their customers could not use the service.
What I'd care more about is whether the people posting the ads wish to be indexed; I'd be surprised if many don't. (I don't know about the legal bearing either.)
Perhaps craigslist could put up a checkbox (like they do with the never-checked "It's okay to contact me about products ...") that says something like:
"I'm okay with people finding this listing through another service."
Perhaps because they don't want people to get to their listings that way even if they want to, and thus don't want your opinion? (Edit: and perhaps they really aren't as interested as they claim in making it easier for buyers and sellers to find each other?)
For that analogy to hold, PadMapper would have to be pulling data from a large number of sources. The value that Google News provides is that it aggregates a very large number of sources. PadMapper does not.
In my city, Craigslist is hardly used for rental listings. Everybody uses kijiji.com, which is available on Padmapper. There are alternative data sources, and they are actually used. Just not where you are.
It's pretty clear PadMapper was BUILT off of Craigslist that CL's poor UI is isn't entire reason for being, and it never would have been successful without CL. It's disingenuous to ignore that by listing "rent.com" as an alternate source.
But that is where the source of the problem is - CL has the largest user base for listing and classifieds on the internet and since they have the monopoly on this they have never felt the need to improve the experience on their site. Now that padmapper has started getting large number of users CL feels threatened and is suing them to shut them down.
This type of comment is common. Not only on HN but on forums in general.
While I agree the web is full of lowlifes engaged in web development, many of them in porn or some other area that appeals to base instincts, I find this comment perplexing. Because it is so subjective, yet it tries to seem objective by focusing on some random criteria.
Google employs a "bot army" to scrape the entire web. So what?
If the comment was something like "I don't like Company X." Or even "I don't like Company X because...", it would make sense to me.
But that is not how this common type of comment goes. Instead it suggests that bot=evil, i.e. any sort of automation or any sort of data collection by anyone other than [your favorite company] is "shady".
That's crazy. IMO.
It's what a company does with the data that matters.
Anyway, I'm not keen on 3Taps because they are not provinding bulk data, only API's that require "developer keys". Why?
Either you are going to democratise data, or you are just another schemer trying to find ways to collect infromation about people, in this case people using "your API".
I don't want API's I want the data. I can make my own interfaces thank you.
I thought it was weird that CL didn't add a Disallow line for padmapper to robots.txt from the start (just from a PR perspective).
But robots.txt has no special legal authority, it's just a convention used to communicate a publisher's intent. I'm pretty sure the C&D letter made it 100% clear that CL did not want Padmapper crawling their site or using their data.
I know, but I thought the existence of robots.txt was why Google is allowed to crawl sites. If a site disagrees with the crawling they can add a robots.txt entry and Google will honor it. It at least shows that you are giving the publisher an option.
I don't like Craigslist because of lock in. Sellers must sell there because they are the biggest market, buyers must buy there because they are the biggest market. Everyone is locked in.
Lock-in is super common(and often unintentional or unavoidable), but it goes against the ideal of equal opportunity and competitive markets that rewards companies for a continuous commitment to quality and innovation.
I feel that craigslist has done nothing wrong except for standing in the doorway when other people who want to innovate and improve things are trying to get through (like PadMapper).
The reason why Craigslist is valuable is because of it's large network. It is obvious that smaller sites like PadMapper want to slowly try and chip away at Craiglist content (see PadLister) after they hopefully gain enough traction to supplant them. Padmapper / Padlister are fine to start their own sites and have users come and post their listings there, but not to copy content from Craigslist after they specifically asked them to stop.
> It is obvious that smaller sites like PadMapper want to slowly try and chip away at Craiglist content
It's not Craigslist's content. It's our content, the general public's. Craigslist is just a caretaker. Right now, it's a caretaker acting in its own interests against the public's.
The general public gave it to Craigslist under readily available terms. The general public is free to also give it to your service, but that's up to them.
This is disingenuous. Craigslist isn't doing the public's will. It's just taking advantage of public apathy. (Much like Microsoft did with the browser wars.)
Yes you did post some facts about your listing to CL, it is also well within Craigslists right to allow others to copy it or not. Just because you supplied the data to CL doesn't mean you can dictate to them how they should run their business.
A copyright clause designed to allow Craigslist to stop abuse shouldn't be used to squash competitors. That's not dictating "how they should run their business." That's telling them to stop being selfish.
Again, disingenuous. CL has the network effect. That's like Microsoft saying, Users are free to install their own browsers. Their current actions are only to preserve that, not to squash abuses. (Which is what they usually do with the power of their TOU.)
Apple's network effect or market position clearly doesn't make other phones and devices a non-starter. Craiglist's network effect and market position clearly makes other housing listing sites a non-starter.
It also does not mean that CL has any obligation to provide that data to another for free. If you want the data in multiple locations, post it to those other locations.
The purpose of CL is to provide data for free. Why should they care how that data is packaged, so long as this packaging is not abusive, but actually beneficial?
The real reason: Craigslist wants to hold onto their monopoly position.
True, but it's also very easy as a buyer (or seller) to use more than one site. Nothing about Craigslist is exclusive and in that sense the lock-in is very weak.
A Craigslist competitor doesn't need a huge market share to be viable. For me as a seller, it just has to expand my audience of buyers enough to justify the small amount of time it takes to post and manage a second listing.
The license offered by craigslist was for a mobile application only. There was nothing offered for websites. You can see this as either, 'license offered with terms' or 'no license offered at all'
How do people not see this for exactly what it is: The same situation as we are finding with cable providers not wanting to be dumb pipes. The only difference is people don't pay money to CL. They just give CL all their information, and CL decides what to do with it.
Craigslist has no divine right to its users' information; it just happens to be the only viable option. If Craigslist didn't exist, someone else would, and would probably do it better.
If Craigslist wants to continue existing, they should take advantage of the fact that they are the go-to for online classifieds, create a modest subscription-based API and let the information flow.
The problem is that Craigslist has a monopoly. They use that monopoly to shut down competitors, resulting in millions of hours of lost time for users every month.
But on the other hand, "users and developers are exasperated with Craigslist’s insistence on preserving an outdated interface and design."
I don't know the legal merits of either party's position, but from an ethical perspective, I tend to side with Craigslist here. Dissatisfaction with a commercial site's UI is not just cause for using their data without permission, particularly if they had made an effort to offer a licensing agreement, whatever the terms might have been.