Authenticated Scraping

Learn how to scrape content behind authentication using cookies

Important: Only use authenticated scraping on systems where you have explicit permission from both parties (yourself and the platform owner), such as internal, self-hosted tools or resources you fully control. Do not use authentication on platforms unless you are certain it abides by the site’s Terms and Conditions and get written permission when in doubt. Using session cookies improperly can violate terms of service or laws; always confirm you are authorized to access protected content in this way.

Overview

The recommended approach for authenticated scraping is cookie-based authentication, where you:

1. Login manually to your application
2. Extract the session cookie from DevTools
3. Use the cookie with Firecrawl to access protected pages

Cookie Expiration Times:

• Internal tools: Often 7-30 days or longer
• Other tools: Often hours or minutes

Internal tools typically have longer cookie lifespans, making this method ideal for recurring scraping tasks.

---

Setup

1. Get API Key

   Get your Firecrawl API key from firecrawl.dev/app

2. Install Dependencies

   npm

   npm install @mendable/firecrawl-js
   

   Node.js < v20: If you’re using Node.js version 19 or earlier, you’ll also need to install dotenv:

   npm install dotenv
   

   And import it with import 'dotenv/config' at the top of your file.

3. Configure Environment

   Create a .env file:

   .env

   FIRECRAWL_API_KEY=your_firecrawl_api_key
   

---

Method 1: Cookie-Based Authentication

Step 1: Extract Cookies from DevTools

Demo Application: You can practice with our demo app at https://firecrawl-auth.vercel.app

• Email: [e-mail]
• Password: password123

1. Login to Your Application

   Navigate to https://firecrawl-auth.vercel.app and login with the credentials above

2. Open DevTools

   Press F12 or right-click → “Inspect”

3. Navigate to Application Tab

   Click the Application tab (Chrome) or Storage tab (Firefox)

4. Find and Copy Cookie

   1. Expand Cookies in the sidebar
   2. Click on your domain
   3. Find the auth-token cookie
   4. Double-click the Value and copy it

   

For the demo app, the cookie looks like:

auth-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiJleGFtcGxlLXVzZXItaWQiLCJlbWFpbCI6InRlc3RAZXhhbXBsZS5jb20ifQ.example-signature-hash

Important: Cookies are sensitive credentials. Never share them publicly or commit them to version control. Treat them like passwords.

Step 2: Use Cookies with Firecrawl

import FirecrawlApp from "@mendable/firecrawl-js";

const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

const result = await app.scrape("https://firecrawl-auth.vercel.app/dashboard", {
  formats: ["markdown", "screenshot"],
  headers: {
    Cookie: 'auth-token=COOKIE_GOES_HERE'
  },
  waitFor: 3000, // Wait 3 extra seconds for the page to load
});

console.log("=== Markdown ===\n" + result.markdown + "\n\n=== Screenshot URL ===\n" + result.screenshot);

Best Practices

Cookie Security

• Store cookies in environment variables
• Never commit cookies to git
• Rotate cookies regularly
• Use .gitignore for .env files

Cookie Expiration

• Check expiration times in DevTools
• Set up alerts before expiration
• Re-extract cookies when they expire
• Consider using form-based auth for short-lived cookies

Rate Limiting

• Respect the application’s rate limits
• Add delays between requests
• Monitor for 429 (Too Many Requests) errors
• Use exponential backoff for retries

Error Handling

• Check for 401/403 errors (expired cookies)
• Validate response content
• Log authentication failures
• Have fallback authentication methods

---

Troubleshooting

Getting 401 Unauthorized Errors

Possible causes:

• Cookie has expired
• Cookie was copied incorrectly
• Application requires additional headers
• Session was invalidated on the server

Solutions:

• Re-extract cookies from DevTools after a fresh login
• Check if you need multiple cookies (session + CSRF token)
• Verify the cookie domain matches your target URL

Cookie Not Working

Check these:

• Is the cookie name correct?
• Did you copy the entire cookie value?
• Does the cookie have the correct domain?
• Is there a Path restriction on the cookie?
• Are there multiple cookies required?

Pro tip: Copy all cookies from DevTools and test with all of them, then remove one by one to find which are required.

Session Expires Too Quickly

For short-lived sessions:

• Use form-based authentication instead
• Automate the login process with actions
• Set up a cron job to refresh cookies
• Consider requesting longer session times from your internal tool’s admin

Cookie Lifespan for Internal Tools: Many internal tools set cookies with 7-30 day expiration times, making them ideal for recurring scraping tasks. Check your cookie’s Expires field in DevTools to see how long it’s valid.