large files

2 posts with the tag "large files"

How to Filter and Organize Cloud Files by Size With Blober

Jul 20, 2026

Why Size Is an Organization Decision

Size changes cost, transfer time, storage choice, and retrieval behavior.

A videographer may want multi-gigabyte masters in object storage and small proxies on a collaboration drive.
An IT team may separate large database dumps from small configuration backups.
A business may need to find oversized attachments consuming expensive shared storage.
A researcher may move large datasets to archive storage while keeping papers and README files easy to access.

Blober exposes size in two different ways: broad source filters and exact destination variables.

Source Size Filters

Select a source folder, then expand Advanced Filters (Optional). File Size offers:

Small: under approximately 1 MB
Medium: approximately 1-10 MB
Large: over approximately 10 MB
Any size

The filter decides which files enter the workflow. It does not rename them or create size folders.

For example, to copy only large videos:

Select the project or media folder.
Choose video extensions.
Choose Large.
Select an archive destination.
Preserve paths or add a template.

This combines type and size rather than assuming every MP4 is large.

Exact Size Variables

Path templates provide:

{file_size}
{file_size_mb}

{file_size} is the exact source-reported byte count:

{file_size_mb} converts it to megabytes rounded to two decimal places:

1.50

A template:

by-size/{file_size_mb}/{filename}

could produce:

by-size/1.50/document.pdf

This does not create useful buckets named small, medium, or large. It creates one folder for each rounded size value. In a varied media collection that can mean thousands of nearly unique folders.

The Practical Way to Build Size Categories

Use one workflow per meaningful size class and put a static category in its template.

Large archive workflow

Size filter: Large
Template: large-files/{file_dir}/{filename}
Destination: low-cost object storage

Medium working-assets workflow

Size filter: Medium
Template: medium-files/{file_dir}/{filename}
Destination: team collaboration storage

Small document workflow

Size filter: Small
Template: small-files/{file_ext}/{file_dir}/{filename}
Destination: searchable document share

This produces human categories and keeps the rule obvious in Preview & Summary.

When Exact Size in the Path Is Useful

Exact size variables still have valid uses:

Evidence or export manifests where visible byte count helps verification
Diagnostic transfers investigating unexpectedly large objects
Temporary audit collections
Comparing provider-reported object sizes

Example:

size-audit/{file_size}/{file_dir}/{filename}

For a permanent library, exact-size directories are usually less usable than size filters plus a static label.

Unknown and Zero Size

Not every provider reports a meaningful size for every object. When size is unknown, Blober renders:

NO_SIZE
NO_SIZE_MB

Therefore:

{file_size_mb}/{filename}

can produce:

NO_SIZE_MB/document.pdf

This is deliberate. A visible marker like this is safer than silently treating an unknown object as zero bytes.

Preview the source before relying on size. If many files show unknown size, organize them by type, date, or relative folder instead.

Real Workflows

Video agency: masters versus proxies

Large size alone does not prove a file is a master, but it is a useful first boundary when combined with names and extensions:

File Types: video
Size: Large
Exclude: **/{proxy,proxies,temp,cache}/**
Template: masters/{file_created_date}/{file_dir}/{filename}

Create a separate workflow for proxy folders if they need a cheaper or shorter-lived destination.

Research lab: datasets versus documentation

Select the study folder.

Large workflow: archive datasets and binaries.
Small/medium workflow: copy PDFs, Markdown, notebooks, and charts to collaboration storage.

The two destinations can serve different access patterns without changing the source.

SME: identify storage-heavy files

Select a shared folder, choose Large, and preview the result. Even before running a transfer, the preview provides a focused list and total known size. Copy candidates to an archive after owners confirm they are no longer active.

Size and Cloud Cost

Size filtering helps decide what to place in a cheaper tier, but it does not calculate provider charges. Storage price, retrieval fees, egress, minimum retention, and API operations still matter.

A sensible workflow is:

Identify large inactive files.
Copy them to a separate archive.
Verify access and integrity.
Retain the original until policy permits deletion.
Use Move only after the archive is trusted.

Learn the safety difference in Copy As-Is, Organize During Backup, or Reorganize With Move, and combine size with type filters for better targeting.

Large Cloud Transfers: When a Desktop App Is the Right Tool

Jun 27, 2026

Moving a few files is easy anywhere. Moving terabytes is a question of where the work runs, and the answer depends on the job. For most transfers, a desktop app on your own machine beats a browser-based or hosted tool. For a single bulk move of a very large dataset into one provider, that provider's own migration service can be faster. This page sorts out which is which.

The Ceiling Inside a Browser Tab

A browser-based transfer happens inside a tab, and a tab is a small, supervised space. It has a memory budget, it can be paused or discarded when you switch away, and it usually holds only a handful of connections. For a dozen photos this is invisible. For a media library it is the wall the transfer hits: the tab slows, the laptop heats up, and a job that should run for an hour cannot stay awake long enough to finish.

The Timeout and the Bandwidth Bill

Hosted services move work onto their own machines, which sounds like an upgrade until you look at the path. The data has to travel from the source into their infrastructure, and then back out to the destination. You wait for both legs, and the company pays for the bandwidth in the middle. That cost does not disappear. It comes back as a subscription, a per-GB charge, or a monthly transfer cap.

Server functions also run on a clock. AWS Lambda, for example, documents a maximum function timeout of 900 seconds, or 15 minutes^[1]. Many serverless jobs are designed to finish in seconds or minutes, not the hours a large migration needs. When the clock runs out mid-file, the job has to be broken into pieces and stitched back together; if that orchestration is wrong, gaps and duplicates are the failure mode.

What a Desktop Process Can Actually Do

A desktop app is not boxed in the same way. It can use the memory and processor your machine actually has, open many transfers in parallel, and keep running for as long as the work takes. If the connection drops, Blober can retry failed records; if you stop and rerun a workflow, skip-existing keeps completed files from being sent again.

Blober also streams rather than stages. Instead of saving a full copy of every file to your disk and uploading it again, it flows the data through and writes it straight to the destination, so disk space stops being the limit. The difference between a streamed transfer and download-and-reupload is small for ten files and decisive for ten thousand, which is covered in Server-to-Server vs Download-and-Reupload.

Your Connection, Your Rules

Because the transfer runs from your machine, it uses your own connection and your own IP address. There is no shared relay address for a provider to throttle or block for every user of the tool at once. A hosted service, by contrast, reaches every provider from the same pool of company servers, which is the traffic pattern most likely to attract shared-service limits. Your own account and API limits still apply either way.

When a Provider's Own Migration Wins

For a single move of a very large dataset, tens of terabytes or tens of millions of files, into one destination, the fastest path is often not your machine at all. Some providers run datacenter-to-datacenter migration, where the data never makes the round trip through your home or office connection. Backblaze, for example, lists assisted data migration as available for B2 Cloud Storage^[2], and what can take weeks streaming through your own link can take far less over a datacenter one. If you are doing one big move into one provider and you can grant that service access, start there. There is a worked 25 TB example in Migrating 100M+ Files from DigitalOcean Spaces to Backblaze B2.

This is not a flaw in any one tool. Any client-side transfer, a desktop app or a command-line tool alike, has to pull the data down to your machine and push it back up, which is two trips through your connection. For the very largest one-time moves, skipping those trips is the whole point.

Where a Desktop App Is the Right Tool

A provider migration helps in a narrow case: one big move, into one provider that offers it, when you can hand that provider access. Outside that case, a desktop app like Blober is usually the better fit, and for some teams it is the only acceptable one.

When you cannot share credentials with anyone. Assisted migration means granting a third party access to your storage. For regulated data, client data under NDA, or anything inside a SOC 2 or ISO 27001 boundary, that is often ruled out. A team in that position cannot hand its keys to a destination provider's migration service any more than it could hand them to a relay. Blober keeps your credentials on your own machine and adds no third party to the path, so the transfer stays inside the controls you already have. See How Blober Works.
When the move is ongoing, not one-time. Migration services are built for a single bulk lift. New uploads, folder moves, periodic refreshes, and re-running a saved job are what a desktop tool is for.
When no assisted path exists. Provider migration almost always moves data into that one provider. Moving between two arbitrary providers, or pulling a library out of a consumer cloud like GoPro Cloud or Google Photos, has no assisted option. A desktop app that speaks each provider directly is the practical route.
When the scale is moderate. For gigabytes or a few terabytes, arranging an assisted migration is more trouble than it is worth. A desktop app just runs.
When you want to watch it happen. A desktop transfer stays visible and under your control, so you can pause, resume, and verify on your own machine.

What a Desktop App Does and Does Not Solve

A desktop app is not magic, and it is not always the fastest option. The bytes travel over your own connection, so a slow link is still slow, and for the very largest one-time moves a provider's datacenter migration can beat it outright. What a desktop app gives you is control and privacy: no relay, no third party holding your keys, and the reach to move between any providers you connect, including the ones with no migration service at all.

Frequently Asked Questions

Is a desktop app the fastest way to move a very large dataset? Not always. For a one-time move of tens of terabytes into a single provider that offers datacenter migration, that route is usually faster, because it skips the trip through your connection. A desktop app wins on control, privacy, ongoing use, and moves between providers that have no assisted option.

What if I cannot give a third party access to my storage? Then a desktop app is the right choice. Blober uses your credentials on your own machine and adds no outside party to the transfer, so it fits inside compliance boundaries that rule out both hosted relays and assisted migration services.

Does the data still go through my computer? Yes. A desktop tool streams the data through your machine to the destination. The point is that it does not also route through a separate company's servers, and it does not save a second full copy to your disk.

What happens if my transfer is interrupted? Blober retries failed records during a run. If you stop and rerun the workflow, skip-existing avoids sending files already written at the destination, so the job continues at file level instead of starting the whole migration again.

Next Steps

For a very large one-time move, read Migrating 100M+ Files from DigitalOcean Spaces to Backblaze B2.
See how the pieces fit in How Blober Works.
Read the mechanic behind direct transfers in Server-to-Server vs Download-and-Reupload.
Ready to move your files? Get Blober.