WP Content Crawler is a WordPress plugin built to collect and publish content from virtually any website. Whether you are aggregating news, building affiliate feeds, creating a testing environment for themes or plugins, monitoring competitors, or assembling curated collections, the plugin manages the entire workflow from inside your WordPress admin panel.
The plugin supports a wide range of use cases. You can build a personal news aggregator that collects posts from favorite sources, scrape products from shopping sites for WooCommerce stores, gather affiliate program listings to earn commissions, collect plugins and themes to build resource libraries, or keep an eye on competitors. The internet is full of content, and this tool helps you organize it automatically.
The built-in visual inspector lets you click any element on a target page to capture its CSS selector instantly. You can also retrieve alternative selectors and test them in real time without leaving the admin dashboard.
After configuration, the plugin locates post URLs and tracks them in the background. New tracking updates generate fresh messages automatically. You can limit how often a post updates, set the update interval, and ignore older posts to keep your site current without redundancy.
Define exactly how often URL collection and post-crawl events should run for each site. For example, save three posts every minute or run URL collection five times every two minutes. Scheduling is fully customizable to match your server capacity and content goals.
Save every post detail: title, excerpt, content, tags, categories, slug, date, custom meta, taxonomies, meta keywords, meta descriptions, featured image, post image, and status. The plugin captures the full dataset from the source page.
Handle paginated articles and posts containing internal lists. Extract list items, apply custom templates to each entry, and reverse list order when needed.
If a destination category does not exist, the plugin creates it automatically—including subcategories—using CSS selectors. You can save slugs from the source site, insert custom text, or build permalink templates with shortcodes. Taxonomy values can be pulled automatically or entered manually.
Assign scraped content to any available post type, including standard posts, pages, products, or custom post types. Save custom taxonomy categories and populate custom meta fields using CSS selectors or manual input.
Prepare templates for post content, titles, excerpts, list items, and gallery items using shortcodes. Each CSS selector has an option box for localization, replacement, calculation, modeling, and JSON parsing.
Use plain text or regular expressions for find-and-replace operations. Modify HTML, create custom elements, write selectors to use them, alter image URLs, and remove unwanted links, ads, or comments by targeting their CSS selectors.
Save all images within post content with a single checkbox. Save images as galleries using custom templates for each item, or save them directly as WooCommerce galleries. The plugin assigns srcset attributes for faster responsive loading and preserves alt and title attributes for SEO.
Rename, copy, and move saved media files easily. Define title, description, caption, and alt text using templates that support shortcodes. Random filenames can also be assigned for organizational flexibility.
Scrape products with price, stock, shipping details, attributes, and advanced options. Save products as simple or external products, set download file options, and mark products as virtual. Compatible with WooCommerce 3.3 and above.
Route requests through proxies when your IP lacks access. Attach session cookies to crawl target sites as if logged in, enabling access to protected or personalized content.
Prevent duplicates by checking URL, title, or content. For WooCommerce, products with existing SKUs are considered duplicates and skipped. Schedule publication by adding or removing minutes from the source post date.
Integrate with Google Cloud Translation, Microsoft Translator, Yandex Translate, or Amazon Translate APIs for automatic translation. Use Spin Rewriter API for automatic content rotation to support SEO strategies.
Test selectors, regex patterns, find-and-replace rules, proxies, and crawl settings in real time. Enable caching for faster testing with fewer requests. The manual crawl tool lets you save posts by URL or category URL and run multiple crawls at once.
Set CSS selectors whose values must not be empty on category and post pages. Receive email notifications when empty values are detected. The plugin displays detailed error messages and handles mixed character encodings.
Transform iframes and scripts into secure shortcodes that display only from allowed source domains. The plugin supports modern PHP versions and works with Chrome, Firefox, Safari, Opera, and Edge.
Monitor active sites, crawled posts, updated posts, and scheduled events from a detailed dashboard. Update the plugin with one click. Access online documentation, quick guides beside each setting, and video tutorials.
Most scraping solutions require custom code or complex configuration. WP Content Crawler replaces that complexity with a visual interface, robust scheduling, and deep WordPress integration—including native WooCommerce support. The GPL license allows unlimited domain usage, and manual lifetime updates keep the tool current.
If you need reliable, automated content collection within WordPress, WP Content Crawler delivers the control, flexibility, and integration required for professional workflows. From visual selection to scheduled publishing and full media management, it covers the full scraping lifecycle without leaving your admin panel.
WP Content Crawler automatically pulls content from virtually any website directly into WordPress. Use the built-in visual selector to target elements, schedule automated crawls, and publish posts, products, images, and custom fields with complete control. Visual CSS selector inspector for precise targeting Automatic background tracking and scheduled updates Full WooCommerce product scraping support Save images, galleries, taxonomies, slugs, and custom meta GPL license with unlimited domain usage
From multipurpose themes to niche templates
