Remaining Websites Extraction Summary
Run Scope
The extraction pipeline processed the five remaining active websites from sources/websites.yaml one at a time. Each crawl used a 25-page cap, remained on the registered domain, respected robots.txt, saved raw and cleaned HTML, generated Markdown and per-page JSON metadata, and recorded deferred URLs and crawl exceptions.
giniinc.tech was not reprocessed.
Results
| Website | Pages processed | Metadata files | Failures | Deferred URLs | Skipped URLs | Redirects | Exact duplicates |
|---|---|---|---|---|---|---|---|
| giniinc.com | 25 | 25 | 0 | 8 | 12 | 2 | 0 |
| eriverspace.org | 20 | 20 | 0 | 0 | 2 | 0 | 2 |
| ellijayriverspace.com | 17 | 17 | 0 | 0 | 2 | 0 | 0 |
| giniverse108.com | 25 | 25 | 0 | 425 | 3 | 1 | 0 |
| foodtechsupply.com | 25 | 25 | 0 | 445 | 2 | 0 | 0 |
| Total | 112 | 112 | 0 | 878 | 21 | 3 | 2 |
All 112 metadata files parsed successfully. Each processed page has one raw HTML file, one cleaned HTML file, one Markdown file, and one metadata JSON file. No <nav>, <header>, <footer>, or <aside> elements remained in the cleaned HTML validation scan.
Crawl Notes
giniinc.com
The cap was reached after 25 pages. Eight discovered URLs were deferred. Two aliases redirected to already processed canonical service pages:
/service/graphics-design→/service/branding/service/data-services-2→/service/data-services
eriverspace.org
The crawl completed naturally at 20 pages with no deferred URLs. Two pages had content identical to /pages/rural-america-global:
/pages/events/pages/group-training
The duplicate pages remain saved and are flagged in their metadata and crawl report.
ellijayriverspace.com
The crawl completed naturally at 17 pages with no deferred URLs or duplicate content.
giniverse108.com
The cap was reached at 25 pages, leaving 425 discovered catalog URLs deferred. The apex-domain homepage redirected to the canonical www hostname and was deduplicated.
foodtechsupply.com
The cap was reached at 25 pages, leaving 445 discovered catalog URLs deferred. The large deferred set is primarily consistent with an ecommerce catalog and should be prioritized before a deeper crawl.
Recommended Extractor Improvements
Before scaling deeper:
- Add sitemap ingestion and URL-type prioritization so core pages, collections, products, articles, and policies can receive separate crawl budgets.
- Add frequency-based boilerplate detection across pages to remove repeated promotional blocks that appear inside page bodies, beyond semantic layout tags and CSS selectors.
- Extract structured data from JSON-LD and ecommerce product markup into dedicated metadata fields such as SKU, price, availability, brand, and category.
- Add near-duplicate detection using normalized text similarity. The current pipeline reliably flags exact content duplicates but not lightly modified template copies.
- Capture full redirect chains and original redirect status codes. The current report records source and final URLs after automatic redirect following.
- Add run manifests and optional stale-artifact cleanup for repeat crawls so removed website pages can be reconciled safely.
The two large ecommerce sites should not simply receive an unlimited cap. A sitemap-driven, category-balanced crawl will produce a more representative knowledge set with less catalog bias.