DataSift Bot

The DataSift Bot is an automated process that we use to expand links mentioned in content from our data sources. For example, when a user mentions a link, or clicks on a link from a URL shortening service such as Bitly, the DataSift Bot will follow this link, resolving it to the final destination URL and retrieving metadata for the shared link.

The DataSift Bot creates minimal traffic as we ensure it only visits each URL a maximum of once a day, caching the result for future mentions. Also, unlike a web crawler, the DataSift Bot does not scrape and follow links within the destination page content, so only visits the shared URL itself. Please let us know if you want to blacklist your domain from our service.