Python Automation Intern (AI, n8n, Web Scraping)
Figmenta · Remote
March 2026 - April 2026
Over two months I built Figmenta's social media data collection and processing pipeline in n8n. One workflow picks the companies to track, scrapes their recent posts on Instagram, TikTok and X with Apify, then parses, validates and stores the metrics in Supabase, with the media in S3 and R2. The workflow logs every step, so a stopped run resumes where it left off, and AI agents score each post with Z-scores so the team can compare engagement across brands and platforms. Each run covers more than 200 company profiles.
In short
Engineered 15+ production-grade automation workflows handling 6,000+ data points per run for large-scale social platform data processing.
Designed multi-stage ETL pipelines that scraped platform data, stored structured outputs, and triggered API-driven downstream workflows.
Integrated database-backed pipelines with AI agents to compute Z-score-based analytics, turning raw data into actionable marketing signals.
Built dynamic workflow chains to provision R2 storage buckets and serve structured media outputs.
- Nodes in one n8n workflow
- 150+
- Company profiles per run
- 200+
- Platforms: Instagram, TikTok, X
- 3
- Per run, once or twice a week
- 6-8 h
03.1How a run works
01 Trigger
Schedule or manual run
02 Select
Unique rows · Company list · Profile lookup
03 Scrape
Scrape profiles · Post loop
04 Parse
Split by format · Parse and validate
05 Store
Metrics table · Media buckets
06 Track and score
Run state · Z-score scoring
03.2After I joined
What changed
- One end-to-end workflow of 150+ nodes, started on a schedule or by hand
- Instagram, TikTok and X for 200+ company profiles in every run
- Parsed metrics in Supabase, media in S3 and R2
- Runs once or twice a week for 6-8 hours and resumes after a failure
- Saves roughly 3-4 hours a run
Problems I solved
- TikTok is blocked in India
- Moved its scraping to offshore servers with the senior engineers.
- Bucket authentication
- Fixed the S3 and R2 authentication that was failing media uploads.
- Expiring media links
- Reworked media collection to save files before each platform's CDN links expire.
- Unreliable scraper output
- Validation drops missing or corrupted records, and a stopped run resumes from its last completed step.
- Parallel writes
- Built a second version that writes to Postgres in parallel and measured it against the original.
- Storage cost
- Stopped storing full scraped payloads. Only parsed metrics go in the database.
