URL Fetching

The URL Fetching feature allows the AI to access and retrieve the contents of a web page before generating a response. This feature is especially useful in scenarios where the AI needs real-time or context-specific data from external sources to enhance its output.

Whether you're working on web scraping tasks, retrieving documentation, or integrating data from online sources, URL fetching can provide the necessary page content (HTML, APIs, etc.) to the AI, making it more effective in generating accurate and tailored results.

Availability: It is available for all users (including the Free plan) across all tools.

Key Use Cases

  1. Web Scraping: URL Fetching can pull the full HTML content from any given public page, allowing the AI to analyze the structure and content. This is useful for tasks like parsing data from websites or automation data extraction processes.
  2. API & Documentation Retrieval: For developers working with libraries, frameworks, or third-party APIs, URL Fetching can retrieve the relevant documentation, so the AI can better understand the context and generate code snippets accordingly.
  3. Dynamic Content Handling: Large Language Models (LLMs) are generally trained on data up to a specific date, so knowledge after that date is not available to the AI. URL Fetching enables it to gather up-to-date information from the web, making it more suitable for situations that require the latest data or documentation.

How Does It Work?

When you provide a URL in your prompt and instruct the AI to fetch it, the AI intelligently decides whether to fetch that URL. If it fetches the page, it processes its content before responding to your query. The AI uses the retrieved data as context, allowing it to create more precise and context-aware responses.

For example, if you ask the AI to scrape a specific product list from an e-commerce page that you entered, or to generate a code snippet based on the documentation of a framework, the URL Fetching feature ensures that the AI has access to the web page content to complete the task effectively.

Example Use Cases

  • Web Scraping: You can instruct the AI to fetch a website, extract HTML, and parse specific elements from a page, like product listings, article data, or stock information, like so: "Fetch the product details from this page: https://books.toscrape.com/ and save all product details in a JSON file"

URL Fetching Feature Use

Make sure to enable URL Fetching before making a request. If you see "the URL has been fetched successfully" message in the result, then the HTML content of the target page is accessible to the selected AI (in this case, GPT-4o Mini) and will, therefore, generate code accordingly.

  • API Documentation: When working on a project that integrates third-party APIs or libraries, the AI can fetch the API documentation and generate the correct code implementation for you, something like this: "Fetch this API doc for me: [Documentation URL] and generate a code snippet to do [your desired task]

Best Practices & Limitations

  • URL Fetching is enabled by default, but it does not mean it will be used everytime. Instead, if you want to use it, you have to specifically instruct the AI to fetch it to do your desired task.
  • Public Pages Only: Ensure that the URLs you provide are accessible in an incognito tab in your browser, that's because pages that requires login or are hidden from certain countries may result to the failure of the retrieval.
  • Avoid Blocking Sites: Be mindful of websites that have restrictions, such as those that block bots, or have anti-scraping measures in place.
  • Some pages with complex JavaScript or dynamic content may not render fully.
  • Currently, only Claude 3.5 Sonnet, GPT-4o, and GPT-4o Mini supports URL Fetching, we will work to add this on more LLMs.
  • Claude 3.5 Sonnet sometimes simulates fetching URLs, while it is not actually doing so, especially in subsequent chat conversations. To avoid this, specifically instruct it to actually fetch it using the get_url_content() function.

This feature enhances the versatility of CodingFleet, allowing it to provide accurate and relevant outputs by accessing real-time web content.

   Last Updated: oct 2024