Web scraping: Considerations on Copyright?
Good evening Gitlabers!
As discussed, I will present the first steps of my project my project on tuesday. At the same time, I will try to give some info about text mining workflows. I ve put some general and basic information the topic of text mining and data into the Wiki. This should be a work in progress. You don’t have to read everything, but if you are interested you can (it’s a wiki)!
Please take a look at the repository as well, where I will write a short description of Tuesday’s lesson!
Since I use the method of web scraping (see Wiki!!), a big/medium/small issue appeared: Please read this blog entry and then the terms of use of “Der Spiegel”.
My question to you would be: What are problems regarding (e.g.) copyright if I would perform web scraping and use aggregated data of texts from the Spiegel-Archive? Or do you have other ideas why this could be problematic?
(I know it’s a short-termed issue, but I would be glad if everybody could take a 5-minute-look at the two links before Tuesdays lesson.) Thank you so much!
Activity
- A deleted user
added Announcement Mutual help labels
Thank you @patrick.gut for this interesting introduction! I've heard of text mining before but never actually had anything to do with it, so it's great to have a project like this in the Übung.
I don't know if we're meant to already put some thoughts online or wait until tomorrow. Either way I'll post my first impressions.
Regarding your question: Legally, if the Spiegel wanted, they'd have a case against you. The paragraph What You Should Know About Copyright and Intellectual Property Rights is making it somewhat clear what their usage guidelines are. So without first discussing your project with them, they could very easily rightfully block or sue you. The blogpost makes it clear that we find ourselves in the realm of Law here, where fairness is irrelevant and interpretation is key.
This example specifically could lead us to a discussion about Law and digital humanities in general. If I remember correctly @martin.dusinberre mentioned the large amount of legal questions he has to answer while creating Lives in Transit. What can be used (pictures, texts, etc.), who can access the data and the underlying code are key questions, which probably arise in most if not all of the projects we've looked at so far. And there's another layer to consider as soon as the output is something interactive like a game or a website, where questions of user data come up.
Thinking about practical legal problems is something, which is - in my opinion - not talked enough about while studying (at least in my experience, I'd be happy to be proven wrong). If there is time, I'd really like to hear more about it from someone who has encountered these questions.
I know this isn't a complete solution, and maybe it isn't even a good one, but I generally base my copyright-related designs on a quick consideration of the interests of the affected parties. Basically, if what you're doing somehow helps the party whose data you're scraping, then they have no good reason to make a case against you. For example, if you made a cool project that showed the richness of Der Spiegel's archives, and didn't release/republish full articles, and if they don't have any published guidelines for researchers/scrapers, I don't see the harm, and therefore assume you're in the clear.
It seems to me like breaking the law isn't what matters, so long as the 'victim' stands to benefit from your work. That said, a quick email to them can be a good idea---if you did so, you'd just have to stress that it's research, that you'll never re-publish what you scraped, and that you won't even quote anything longer than a paragraph. This all changes, however, if the org actually provides services like (e.g) an API for harvesting data from their site. If they have an API, then you should definitely follow whichever guidelines are available.
NYT for example published a 'NYT Annotated Corpus', 1.8 million articles and metadata in XML format. From this you can assume they have given some thought to the use of their content in research projects, and therefore you should follow the published guidelines.
Finally note, some websites actually have a file that tells bots (search engines for example) how to behave on their site: https://www.spiegel.de/robots.txt . That may give you some clues. I'm no expert in interpreting
robots.txtfiles unfortunately.@patrick.gut Great that you've raised these questions. I'm definitely more cautious than @daniel.mcdonald , based on my LiT experience: I think you should work on the assumption that if a lawsuit could be brought against you, it will be brought against you, as questions of 'victims', and who 'benefits', look quite different in a law court from how they might in the common-sense world. I can talk through a couple of our LiT experiences in 2017-18 (before Danny joined the team) tomorrow, if that would help. I am completely with @henrik.jochum that these issues need to be taught more. They are a fundamental aspect of what it means to do history 'in a digitized world' and we're all going to have to deal with them more extensively in the future.
Maybe of interest: There was a big controversy among the users of the social media platform Mastodon a few months ago, because researchers at the University of Milan scraped and published everyone's posts for a research paper. This was extra egregious because of the culture of Mastodon and its architecture (I posted an explanation here ). The research itself was quite wrongbad, but it is an interesting example of how to conduct research in communities that we are not ourselves part of, and how there might be unspoken rules of conduct that don't seem to follow outsider common sense.
The paper is this one: Mastodon Content Warnings: Inappropriate Contents in a Microblogging Platform
Here is a commentary on it by an academic and Mastodon user: Internet Research is Hard
mentioned in issue #31 (closed)
