Online Historical Obituary Data Collection With Web Scraping Solution

A web scraping solution to automate the process of collecting and processing historical obituary data across a large number of public digital newspaper archives and open-source sites.

DỊCH VỤ CUNG CẤP: Historical Data Web Scraping Solution

Online Historical Obituary Data Collection With Web Scraping Solution

BUSINESS CHALLENGES

Our Client

DIGI-TEXX’s client is the world’s leading provider of family history and genealogy services, with its main office located in the US. As part of the company’s 20 years of collecting, indexing, and digitizing efforts, they currently manage almost 7 billion records. These records include immigration, military service, marriages, and much more.

Project Challenges

Tracing The Historical Profiles

When it comes to researching genealogy, the obituary is a treasure trove of crucial data. Beyond basic biographical details like name, birth, and death dates, they provide insights into geographical locations, relatives’ names, and other important data that may be difficult to find in other historical sources.

Unveil Historical Obituary Data With Web Scraping Solution

Nevertheless, collecting and processing obituary data remains a challenging task for our clients due to several factors:

  • Handling diverse and enormous data sources: A vast amount of historical obituary data is scattered across millions of digital resources from public newspapers, libraries, governments, churches, universities, and funeral home websites,…
  • Manual Inefficiency: Manually extracting and indexing different formats, data types, and complex web structures is time-consuming and costly.
  • Data Duplication: One piece of information can be stored on various resources, as a result, cleansing these data requires time and workforce.
  • Data quality assurance: Ensuring data accuracy, completeness, and consistency can be challenging due to errors, unstructured data, and missing information

Project Scope

The project aims to develop a robust solution that assists clients in automatically collecting historical obituary data across digital resources. Then, the collected data will be standardized to ensure consistency and quality of information.

  • Volume: 450,000 records per URL with 60 URLs/month
  • Extracted fields in each record include a person’s name, gender images, birthplace, age, residence, date of death, location, cause of death,..

SOLUTION

Historical Data Web Scraping Solution

To address these challenges, DIGI-TEXX developed a sophisticated web scraping solution to automate the process of collecting and processing historical obituary data across a large number of digital public newspaper archives and open-source sites. This would enhance the database, providing their users with access to millions of new records. 

Online Historical Obituary Data Collection With Web Scraping Solution
  1. Scraper Development: Our team has integrated machine learning algorithms and natural language processing (NLP) to develop a robust web scraper with the capacities of:
    • Accessing public newspaper archives and online websites including schools, hospitals, churches, etc.
    • Navigating through a vast array of sitemaps, sections, and search results
    • Collecting the necessary data fields from a targeted website, including the person’s name, birthplace, age, date of death, residence, cause of death,…and images.
    • Consolidating collected data into a central repository ensures one record per unique individual.
    • Handling various data formats and structures (PDF, HTML, images)
  2. Data Validation: The collected data was cleaned, standardized, and formatted to match the Client’s database structure.

BUSINESS OUTCOME

  • Optimized data processing time:
    • 20-30 minutes for text-used URL 
    • 2-3 days for more complex URL
  • Expanded Database: Delivered over 450,000 records per URL, significantly expanding the client’s database and providing users with access to millions of historical records.
  • Improved Data Quality And Accuracy: Achieved an accuracy rate of 95%, ensuring reliable and accurate obituary data.
  • Power Machine Learning and AI Applications: Create large datasets for training machine learning models to improve accuracy and performance
Unveil Historical Obituary Data With Web Scraping Solution

CÁC DỰ ÁN LIÊN QUAN

Data Annotation and Labeling Social Media Data To Predict The Pandemic

Data Annotation and Labeling Social Media Data To Predict The Pandemic

DIGI-TEXX provided a robust text annotation service with human-in-the-loop, which combined the power of machine learning, natural language processing (NLP)...

Historical Obituary Data Collection With Web Scraping Solution

Online Historical Obituary Data Collection With Web Scraping Solution

A web scraping solution to automate collecting and processing historical obituary data across public digital newspaper archives and open-source sites.

Data Extraction Solution For Customer Onboarding v6

Giải pháp Trích Xuất Dữ Liệu Tự Động Hỗ Trợ Quy Trình Onboarding Khách Hàng Trực Tuyến

Khách hàng của chúng tôi là công ty dịch vụ tài chính và bảo hiểm quốc tế hàng đầu với hơn 1,5 triệu khách hàng đang hoạt động tại Châu Á, Canada và Hoa Kỳ.

CHÚNG TÔI SẼ GIẢI QUYẾT CÁC KHÓ KHĂN CỦA BẠN