Skip to main content

​
create-folder-structure.sh

Creates local folder structure and optionally downloads files from Azure Blob Storage.

​
Description

This script reads blob paths (typically from list-media-files.sh) and creates a matching local folder structure. It can create placeholder files, download actual files in parallel, or just create directories.

​
Prerequisites

  • Azure CLI installed and configured
  • Authenticated Azure session (az login)
  • Input from list-media-files.sh or similar format

​
Configuration

Edit these variables in the script: 
ACCOUNT_NAME="testdinostr"
CONTAINER_NAME="docs"

​
Usage

​
Basic Syntax

./scripts/list-media-files.sh | ./scripts/create-folder-structure.sh [BASE_DIR] [MODE] [PARALLEL]

​
Parameters

  1. BASE_DIR (optional, default: media-files)
    • Directory where structure will be created
  2. MODE (optional, default: placeholder)
    • placeholder - Create empty placeholder files
    • download - Download actual files from Azure
    • skip - Create only directories, no files
  3. PARALLEL (optional, default: 10)
    • Number of parallel downloads (only for download mode)

​
Examples

​
Create Placeholder Files

./scripts/list-media-files.sh | ./scripts/create-folder-structure.sh

​
Download Actual Files

./scripts/list-media-files.sh | ./scripts/create-folder-structure.sh media-files download

​
Download with High Parallelism

./scripts/list-media-files.sh | ./scripts/create-folder-structure.sh media-files download 20

​
Create Only Directories

./scripts/list-media-files.sh | ./scripts/create-folder-structure.sh media-files skip

​
Custom Directory Name

./scripts/list-media-files.sh | ./scripts/create-folder-structure.sh my-downloads download

​
Output

​
Placeholder Mode

Creating folder structure in: media-files
Mode: placeholder

Found 156 files

Creating placeholder files...
Created: docs/ai-insights/emerging-failures.webp
Created: docs/ai-insights/key-metrics.webp
...

✓ Complete!
  Total files: 156
  Location: media-files

​
Download Mode

Creating folder structure in: media-files
Mode: download
Parallel downloads: 10

Fetching storage account key...
Storage key retrieved successfully

Found 156 files

Starting parallel downloads...

✓ Downloaded: docs/ai-insights/emerging-failures.webp
✓ Downloaded: docs/ai-insights/key-metrics.webp
✗ Failed: docs/some-file.webp
...

✓ Complete!
  Total files: 156
  Location: media-files

​
Features

  • Parallel Downloads: Downloads multiple files simultaneously for speed
  • Smart Fallback: Uses GNU parallel if available, otherwise uses background jobs
  • Progress Tracking: Shows real-time progress with ✓ and ✗ markers
  • Error Handling: Creates placeholder for failed downloads
  • Flexible Modes: Choose between placeholder, download, or directory-only

​
Performance

​
Download Speed

  • Default: 10 parallel downloads
  • Recommended for fast connections: 15-20 parallel
  • Recommended for slow/unstable connections: 5 parallel

​
Optimization Tips

# Fast connection
./scripts/list-media-files.sh | ./scripts/create-folder-structure.sh media-files download 20

# Slow connection
./scripts/list-media-files.sh | ./scripts/create-folder-structure.sh media-files download 5

​
Error Handling

​
Failed to Fetch Storage Key

Failed to fetch storage account key. Falling back to placeholder mode.
Solution: Run az login and verify access to storage account

​
Failed Downloads

Files that fail to download will:
  1. Show ✗ Failed: path/to/file
  2. Create an empty placeholder file
  3. Continue processing other files

​
Command Line Too Long

The script handles this automatically by using background jobs instead of xargs when needed.

​
Troubleshooting

​
No Files Created

  • Verify input format matches expected pattern (contains → https://)
  • Check Azure credentials: az account show

​
Slow Downloads

  • Reduce parallel count: use 5 instead of 10
  • Check network connection
  • Verify Azure region latency

​
Permission Denied

chmod +x scripts/create-folder-structure.sh

​
Advanced Usage

​
Resume Failed Downloads

# First run with placeholder mode
./scripts/list-media-files.sh | ./scripts/create-folder-structure.sh media-files placeholder

# Then download only missing files (manual approach)
find media-files -type f -size 0 | while read f; do
  # Download logic here
done

​
Selective Download

# Download only images
./scripts/list-media-files.sh | grep "\.webp\|\.png\|\.jpg" | ./scripts/create-folder-structure.sh images download

# Download only videos
./scripts/list-media-files.sh | grep "\.mp4" | ./scripts/create-folder-structure.sh videos download