API

API, or Application Programming Interface, allows access provided by the host, enabling direct acquisition of information from the client or application.

Still unclear? Let's break down the use of APIs in simple terms.

When should you consider using an API for your aggregator website? Here’s a checklist:

• You possess the technical skills to manage API connections.
• The news source provides a public API service.
• The API supplies the news feeds necessary for your website.
• You are not compiling data from numerous sources.

Not all sources provide APIs, and often, the information available is limited. Given that each API is offered by different entities, connecting to them varies. If you are sourcing data from 50 different publications, you would need to establish and maintain 50 separate data pipelines, which is quite a task. However, if you have a development team focused on data collection, this could be a viable option.

Web Scraping

In contrast to APIs, web scraping involves extracting data directly from HTML files.

Since you're accessing data embedded in HTML source code, you are not restricted by the host. Essentially, whatever you can view in a browser can be obtained through web scraping.

This is crucial for a news aggregator—getting the news!

I won't delve into specific programming languages like Python or Node.js for web scraping, as they can be complex. Creating scripts for web scraping requires significant skills and effort in both the development and maintenance of scrapers. Instead, I want to share a no-code solution using a tool like Octoparse. It simplifies the scraper creation process and alleviates many challenges faced when developing your own solution.

Investing a week or two to familiarize yourself with its interface and workflow is advisable, enabling you to create your own web scrapers. For a news aggregator website, frequent data updates are necessary. Features such as task scheduling for automated data scraping and database integration can significantly ease your workload.

> Sign up here for a 14-day trial, and Octoparse's support team will guide you throughout the process.

Shallow Learning: From Manual to Automated

Initially, news was categorized manually. Publishers would sift through numerous articles, identifying the relevant ones and categorizing them accordingly.

This manual process is slow and prone to errors. With advancements in machine learning and NLP, automated solutions for news classification have emerged.

From the 1960s to the 2010s, shallow learning techniques dominated text classification, including models like Naive Bayes (NB) and K-nearest neighbor (KNN). Data scientists defined the features, and if done correctly, the algorithm could predict news categories based on those features.

Note: By Christopher Bishop, a **feature* is defined as an individual measurable property or characteristic of a phenomenon being observed.*

Deep Learning Approaches

Since the 2010s, deep learning models have gained prominence (like CNN, GCN, ReNN), and they are now more commonly used for text classification in NLP applications than shallow learning models.

Why is this the case?

The primary distinction between deep learning and shallow learning is that deep learning methods can automatically learn features directly from data, whereas shallow learning relies on human-defined features.

Deep learning methods are not inherently superior to shallow learning models. The choice of method should align with your dataset and the classification goals for the text.

Case Study: Building a News Aggregator from Scratch

Review GDPR Compliance

The General Data Protection Regulation (GDPR) is a data protection law enforced by the EU. Exercise caution when scraping personal data from EU residents.

“‘Personal data’ refers to any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person can be recognized directly or indirectly, particularly through an identifier such as a name, identification number, location data, online identifier, or one or more factors specific to the individual’s physical, physiological, genetic, mental, economic, cultural, or social identity.”

If you are scraping personal data from EU citizens, ensure you have a lawful reason, such as obtaining consent or having a signed contract. Alternatively, you may argue that your actions are in the public interest.

Ensure Compliance with U.S. Copyright Law

When scraping data owned by a U.S. citizen or entity, be mindful of fair use provisions. The law outlines four aspects:

1. The purpose and character of the use, including whether it is commercial or for nonprofit educational purposes.
2. The nature of the copyrighted work: Using more creative works (e.g., novels, movies) is less likely to support fair use than factual works (e.g., technical articles, news items).
3. The amount and significance of the portion used in relation to the entire copyrighted work.
4. The effect of the use on the potential market or value of the copyrighted work.

Some web scraping projects exist in a legal gray area, making it challenging to provide a definitive answer. Many factors influence legality, and exploring historical case studies can provide further insights into this matter.